Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..................................X..] duration: 2.805014148s FAIL (2025-07-09 21:46:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.805041435s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 31.144601151s FAIL (2025-07-09 21:46:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.144626804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................................X.X.....X.......X.........] duration: 41.668995114s FAIL (2025-07-09 21:46:49 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 41.669024599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.X.....X.......X........] duration: 50.609911949s FAIL (2025-07-09 21:46:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 50.609950484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..................................X.] duration: 2.605755043s FAIL (2025-07-09 21:45:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.605994985s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 18.695006267s FAIL (2025-07-09 21:45:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.69502756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................................X.X.....X.......X.......] duration: 37.457792278s FAIL (2025-07-09 21:45:11 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 37.457823664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 33.653548802s FAIL (2025-07-09 21:45:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.653567077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..................................X] duration: 30.001411108s FAIL (2025-07-09 21:45:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001435797s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 31.806774194s FAIL (2025-07-09 21:44:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.806799683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.X.....X.......X......] duration: 24.328909237s FAIL (2025-07-09 21:44:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 24.328963008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.X.....X.......X.....] duration: 15.501390248s FAIL (2025-07-09 21:44:05 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 15.501411945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 50.622430535s FAIL (2025-07-09 21:44:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.622448306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.635968605s FAIL (2025-07-09 21:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.635991359s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.X.....X.......X....] duration: 37.638629507s FAIL (2025-07-09 21:43:49 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 37.63865963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.043990868s FAIL (2025-07-09 21:43:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.044017113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.X.....X.......X...] duration: 36.628509684s FAIL (2025-07-09 21:43:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 36.628541693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.998262475s FAIL (2025-07-09 21:42:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.998290678s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 26.759210792s FAIL (2025-07-09 21:42:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.759238814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.798936268s FAIL (2025-07-09 21:42:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.798968292s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.X.....X.......X..] duration: 27.11842222s FAIL (2025-07-09 21:42:05 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 27.118455106s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 40.410719469s FAIL (2025-07-09 21:41:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.410746461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.588558143s FAIL (2025-07-09 21:41:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.588588447s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.667241657s FAIL (2025-07-09 21:40:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.667266728s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 30.584445026s FAIL (2025-07-09 21:40:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.584468228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.X.....X.......X.] duration: 15.666773718s FAIL (2025-07-09 21:40:52 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 15.666821077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X..........................................X.X.....X.......X] duration: 1m0.003083782s FAIL (2025-07-09 21:40:37 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== ad=317059]: Got next segment. Inline: false Jul 9 21:39:37.379940998 (396h28m37.263354631s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:39:37.379950180 (396h28m37.263363814s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:37.581511580 (396h28m37.464925193s) [seg_pos={0 0}, upload=317059]: Begin upload segment... Jul 9 21:39:37.581575434 (396h28m37.464989047s) [seg_pos={0 0}, upload=317059]: Done begin upload segment. Jul 9 21:39:37.581608674 (396h28m37.465022313s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.581625259 (396h28m37.465038894s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582205955 (396h28m37.465619593s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582226329 (396h28m37.465639963s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582706977 (396h28m37.466120591s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.582754275 (396h28m37.46616791s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583166819 (396h28m37.466580453s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583619396 (396h28m37.467033031s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.583774378 (396h28m37.467188015s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584023120 (396h28m37.467436756s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584314386 (396h28m37.467727999s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.584647353 (396h28m37.468060989s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585042256 (396h28m37.468455885s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585393807 (396h28m37.468807437s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.585718047 (396h28m37.469131684s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586211315 (396h28m37.469624953s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586583666 (396h28m37.469997301s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:37.586947894 (396h28m37.470361527s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Uploading piece... Jul 9 21:39:38.176690504 (396h28m38.060104117s) [upload=317059]: Next returned nil segment Jul 9 21:39:38.176710781 (396h28m38.060124394s) [upload=317059]: Waiting for error group managing segments... Jul 9 21:39:38.299966905 (396h28m38.183380542s) [noise=true , address=77.90.31.17:4773 , piece_id=ZTMXELKUKJ7HAKFHUYD7ON6Z26H7M26TTZJMIVMMFWBWYC7QHHXA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.301708854 (396h28m38.185122474s) [noise=true , address=77.90.31.14:4673 , piece_id=LFWT3VUTU4JAWSLG37JVDPFJ6MHQHUNU7HIMKUS6RHAHMUJLDYPQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.310678123 (396h28m38.194091753s) [noise=true , address=77.90.31.12:4594 , piece_id=XDBGCD5MCJCR7UFBFAZSNDX235ZURWA4PFHDUAF7CJTNPE2J7PLQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.313776483 (396h28m38.197190133s) [noise=true , address=77.90.31.13:4634 , piece_id=KMNACWQZCINWWI2P3VNLKA565AYC7WP457ANBJ3AOH4NOGGAP27Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.323851883 (396h28m38.207265494s) [noise=true , address=77.90.31.21:4909 , piece_id=Z7OGPXQ345RV2XXPRRYEI3MDSNO42LHKNMVBSCBO47DBG2CFAUBQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:38.508194899 (396h28m38.391608539s) [noise=true , address=79.127.220.101:4042 , piece_id=MTJ6QPBBQKMHSMRS5AWEI3CCV26MDV2Q7HYMEJFHIXZUIZPW6MLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:39.206881385 (396h28m39.090295021s) [noise=true , address=79.127.205.246:4254 , piece_id=4TGZP55MUE2EYSI4YQ5SDAIPSTKQKEVZ4FGK32AHOHAHEGXN6AAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:49.770414492 (396h28m49.653828104s) [noise=true , address=104.36.82.13:4198 , piece_id=LCZWBN6S6YXGSZOL3IEGS2RK7EUOGAXU2YUR6NUN2SS45GUDTMLA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:51.357190557 (396h28m51.24060419s) [noise=true , address=79.127.219.38:4019 , piece_id=TB5IGJRG4GD7NKKYCI5VYRZTCIE26PNZ6NEFCAJF5QSGUZA7XYWA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:39:59.059221610 (396h28m58.942635224s) [noise=true , address=104.36.82.7:4927 , piece_id=ZBPZVD57SR4MWWWRGF2JPVJNS3W4ONHV2KXS2WRGXAWNBXBTEMCQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:14.989732200 (396h29m14.873145814s) [noise=true , address=104.36.82.19:4411 , piece_id=TLLLO6OXFPLNVGGDVMC3OZZ3RBL7FRGRE3NYBWCJNXA5WE4ZD7EQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:19.437261406 (396h29m19.320675023s) [noise=true , address=104.36.82.21:4508 , piece_id=D7F6S6MUJZU5HQXKP7HGTLUVE3SNYGMDNJXDX4LDJOUWIK4NKPEA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:21.442370228 (396h29m21.325783845s) [noise=true , address=104.36.82.5:4867 , piece_id=X4DPTHSB5S5HIVI2MBBKLRHKA2WB3X5QORHN3YT6EEI2E7FP4NJQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515903750 (396h29m34.399317379s) [noise=true , address=104.36.82.34:4569 , piece_id=O57ZEE6IQOHWSIF4M5WW5RJ3QH63TDYQJQL7FCIQDGIF3RQD5V2A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: <nil> Jul 9 21:40:34.515927208 (396h29m34.399340828s) [seg_pos={0 0}, upload=317059]: Segment reached optimal threshold of 14 pieces. Jul 9 21:40:34.516037703 (396h29m34.399451328s) [noise=true , address=45.91.95.5:4006 , piece_id=INII2Z7EJBG42JMMJVNDE2NSPMLGDX2GFSGPXQVM7JNKOU4XFF2Q, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516110939 (396h29m34.399524566s) [noise=true , address=45.91.95.3:4019 , piece_id=7QBSLBTSYUR6SKUSGNPPFU47URSU3RI7YCYKHKGQ3W252ECNN2TQ, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516592515 (396h29m34.400006136s) [noise=true , address=45.91.95.4:4023 , piece_id=FDVLTCUYPHSVI4BPM5CBH5WAZWPMOEVFQR3X2BK6AEXBAX5PUQAA, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c): protocol: expected piece hash; context canceled; EOF Jul 9 21:40:34.516693112 (396h29m34.400106729s) [noise=true , address=45.91.95.2:4033 , piece_id=LCDYTFI3LBSXS47JJAD2ID2OBYJVRKR6OL7G63VP3GUF5E6Y643A, seg_pos={0 0}, upload=317059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP): protocol: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33148->45.91.95.2:4033: use of closed network connection Jul 9 21:40:34.516713577 (396h29m34.400127204s) [seg_pos={0 0}, upload=317059]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:40:34.516795229 (396h29m34.400208846s) [upload=317059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:40:34.516800737 (396h29m34.400214368s) [upload=317059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:40:34.680902271 (396h29m34.56431589s) [upload=317059]: Done uploading segments... Jul 9 21:40:34.680921551 (396h29m34.564335167s) [upload=317059]: Upload finished. err: <nil> Duration: 1m0.003109877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001214786s FAIL (2025-07-09 21:40:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001243831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.860772428s FAIL (2025-07-09 21:39:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.860976424s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X..........................................X.X.....X.......] duration: 34.711571261s FAIL (2025-07-09 21:39:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.711600194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.815251012s FAIL (2025-07-09 21:39:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.815277905s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X..........................................X.X.....X......] duration: 32.39091597s FAIL (2025-07-09 21:38:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.390957072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.773471366s FAIL (2025-07-09 21:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.773499906s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.714278827s FAIL (2025-07-09 21:38:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.714313108s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 8.788752119s FAIL (2025-07-09 21:37:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.788781644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X..........................................X.X.....X.....] duration: 25.107909133s FAIL (2025-07-09 21:37:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.107935115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X..........................................X.X.....X....] duration: 43.613900773s FAIL (2025-07-09 21:36:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 43.613932599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.51713679s FAIL (2025-07-09 21:36:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.517165022s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 47.556085433s FAIL (2025-07-09 21:35:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.556107091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X..........................................X.X.....X...] duration: 19.14143085s FAIL (2025-07-09 21:35:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.141491876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.629933706s FAIL (2025-07-09 21:35:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.629988909s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 10.435286133s FAIL (2025-07-09 21:35:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.435316895s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X..........................................X.X.....X..] duration: 38.604113605s FAIL (2025-07-09 21:35:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.604168984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 30.121013561s FAIL (2025-07-09 21:34:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.121028855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.798494519s FAIL (2025-07-09 21:34:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.798524491s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 33.335648912s FAIL (2025-07-09 21:34:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.335666411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X..........................................X.X.....X.] duration: 13.995684057s FAIL (2025-07-09 21:34:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.995715606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X..........................................X.X.....X] duration: 1m0.112083499s FAIL (2025-07-09 21:33:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=316974]: Done starting upload Jul 9 21:32:57.370575308 (396h21m57.253988931s) [upload=316974]: Uploading segments... Jul 9 21:32:57.370624760 (396h21m57.254038388s) [upload=316974]: Got next segment. Inline: false Jul 9 21:32:57.370673795 (396h21m57.254087411s) [upload=316974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:57.370679136 (396h21m57.254092763s) [upload=316974]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:57.582961148 (396h21m57.466374785s) [seg_pos={0 0}, upload=316974]: Begin upload segment... Jul 9 21:32:57.583022579 (396h21m57.466436214s) [seg_pos={0 0}, upload=316974]: Done begin upload segment. Jul 9 21:32:57.583066328 (396h21m57.466479962s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583103738 (396h21m57.466517384s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583157233 (396h21m57.46657086s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583560552 (396h21m57.466974177s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583651559 (396h21m57.467065186s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.583933450 (396h21m57.467347144s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584150169 (396h21m57.467563795s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584328716 (396h21m57.467742351s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584464429 (396h21m57.467878058s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584595391 (396h21m57.468009021s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584764386 (396h21m57.468178012s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.584837755 (396h21m57.468251382s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585097030 (396h21m57.468510642s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585147439 (396h21m57.468561075s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585520264 (396h21m57.468933894s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.585841424 (396h21m57.469255051s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.586208413 (396h21m57.46962205s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:57.590339363 (396h21m57.473752992s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Uploading piece... Jul 9 21:32:58.165062836 (396h21m58.048476449s) [upload=316974]: Next returned nil segment Jul 9 21:32:58.165969382 (396h21m58.049383015s) [upload=316974]: Waiting for error group managing segments... Jul 9 21:32:58.306085244 (396h21m58.18949886s) [noise=true , address=77.90.31.10:4547 , piece_id=OS57KUS7LESOG6BRKAOGEWG6UI6TTFSOVSKZRUJKISRMVVZSPXAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.308664264 (396h21m58.192077877s) [noise=true , address=77.90.31.14:4678 , piece_id=UTQ6GVLIG4CUA2NHXPY3ESFNMVJJ46TG5FPK45AZLRB5LWF3YQXQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.315849833 (396h21m58.199263453s) [noise=true , address=77.90.31.12:4595 , piece_id=U2LGKBA4R23EP2J3DHGUX3L74P74RO6BDHGV4SICA4MLQJB3RJIA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.318417713 (396h21m58.201831348s) [noise=true , address=77.90.31.13:4645 , piece_id=WMS4GMEQTY7MNIZYE6VB3UVPNVCJTN4Y4LVXJJYY2KEQJVU2WIZQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.320523147 (396h21m58.203936767s) [noise=true , address=77.90.31.11:4582 , piece_id=HV7BLYUI434S2KAVC76BTZCPVKLMRF2MENCUGDGG62FJFK62NDAA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.321753883 (396h21m58.205167521s) [noise=true , address=77.90.31.19:4875 , piece_id=FWXOLO72EBUR5OLHGMFAC4QXRSIJCSONADRYDLS7DVC7DG5IY7QQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.324969112 (396h21m58.208382738s) [noise=true , address=77.90.31.15:4702 , piece_id=THY2G2JT577WZBETVEJK7MXXGJG4VPIHKHE3VLM35GOMJJZUPRIQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.332389677 (396h21m58.21580329s) [noise=true , address=77.90.31.17:4783 , piece_id=ENRKOZEDYM2FSEFMDAUGKE3STXJWTLYI5UBJ5E4XBYDR7QIKWQLA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:32:58.339708627 (396h21m58.223122244s) [noise=true , address=79.127.219.38:4008 , piece_id=UQZXI3XTLDWKKNYG5ZKQI4TEX7COKXDNCVUOFQS3B3DTAID433UQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:01.299803153 (396h22m1.183216783s) [noise=true , address=45.91.95.4:4036 , piece_id=OIIA3AHZUSVJAMMSQB6I3DNARIV6LE3RZ47EF6QDMZ6DQPLU525A, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:01.812741897 (396h22m1.696155537s) [noise=true , address=45.91.95.2:4011 , piece_id=BHXWXLISIOQIICHCASQSGKBMM6LG6C4QVBYYMF7NIVLMVMH3COYQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12wJhd32qcV8CYfpYRGgyENLRjYYvw2i5P7pRbiubnQBLocXqgZ, address:45.91.95.2:4011): protocol: storage node overloaded, request limit: 2000 Jul 9 21:33:05.094955074 (396h22m4.9783687s) [noise=true , address=104.36.82.13:4192 , piece_id=XSG3LK7WIXYNCKCGRVA53XTIPVMHDRJQ6223CDUAWDRQWKP27G6Q, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:14.973892194 (396h22m14.857305805s) [noise=true , address=104.36.82.39:4736 , piece_id=ZUJ2T7TTZVKVWF5T3RXBB6F5L4SADEJK4FRZQ4LV5WRXRVUYRDPA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:29.151853866 (396h22m29.035267479s) [noise=true , address=45.91.95.5:4035 , piece_id=A3P3DPU76VDXE3Q4FR7PGYA3S62CM75OPLNVGFW2OAGWBDEQERNA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:41.456622333 (396h22m41.340035967s) [noise=true , address=104.36.82.19:4413 , piece_id=EXJX3MWKUVEQIULSWBB6WP7UYG6RTGYGMWHMC2FSACRQSXNBODJQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: <nil> Jul 9 21:33:57.102214004 (396h22m56.985627621s) [noise=true , address=45.91.95.3:4016 , piece_id=EETP46QZCGNWPFN7D3DJ6VGQDH3V4FSXPYLIN7ZBUYQSPPL2THOA, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12W2ReeeZYTsv172dnPdpy1trUW5UtjHvXz8tDXH1CG3bSpGUiv, address:45.91.95.3:4016): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102257504 (396h22m56.985671132s) [noise=true , address=104.36.82.36:4649 , piece_id=6BL43746OKP7KMIQMDSNIOWVHVMYE5ERVDJMHZ7CWPYB2HRBBRCQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:12Ld1wZ9sxQVpKNNvPFs9oyGhxCPips2YCbq5FiPFwfmuP7U2bY, address:104.36.82.36:4649): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:57.102462044 (396h22m56.985875666s) [noise=true , address=104.36.82.38:4702 , piece_id=5BGIA54ZJJI6P6UY5LMM4MTVLBNUUGJQ7KLBSSJJSZVBK4O7A7KQ, seg_pos={0 0}, upload=316974]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58002->104.36.82.38:4702: use of closed network connection Jul 9 21:33:57.102487809 (396h22m56.985901437s) [seg_pos={0 0}, upload=316974]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:57.212325437 (396h22m57.095739071s) [upload=316974]: Done uploading segments... Jul 9 21:33:57.212373720 (396h22m57.095787346s) [upload=316974]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.112113376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 57.026100645s FAIL (2025-07-09 21:33:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.026113602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.643372442s FAIL (2025-07-09 21:33:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.643393148s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)