Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.41451119s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:45:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.414544105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.354391501s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:44:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.35452248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.467401672s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:44:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.467430456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.489943382s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:43:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.489965491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.411955207s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:43:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.41198351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.381731972s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:42:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.38175917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.49450265s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:41:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.49469786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003536717s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:40:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 30.003564081s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.778664712s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:42:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.77869045s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.944891572s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.944926151s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.782781999s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:41:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.782810233s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.77282873s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:40:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.772922785s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.007136129s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:39:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.007175009s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.746997079s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:38:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.747021468s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.758045766s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:37:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.758072037s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.12630949s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.126338198s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.797597164s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:35:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.797677107s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.063675657s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:35:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.064081219s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.007131655s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:34:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.007162633s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.933010184s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:33:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.933031295s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.004083008s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:33:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.004105885s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.6626737s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:32:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.662706134s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.96078182s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.960805117s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.931994322s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:31:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.932539535s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.742699474s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:30:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.742724951s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.935804326s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:29:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.93583873s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.790873662s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:28:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.790910985s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.919484019s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:28:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.919522062s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.091479511s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:27:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.091521883s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.739335926s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:26:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.739510932s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.853579337s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:25:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.853736807s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.867085364s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:25:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.867109004s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.980054043s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:24:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.980138605s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.889428419s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:24:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.889482565s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.834269726s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:23:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.834330497s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.911131913s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:22:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.911164188s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.043364794s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:21:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.043418094s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.906931988s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:20:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.906959986s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.887467027s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:20:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.88749408s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.887882678s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.887909484s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.028831158s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:18:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.028862287s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.026380253s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:17:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.026403106s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.98257917s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.982613743s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.900402928s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:16:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.900425979s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.150773664s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:15:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.150872496s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.847502408s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:13:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.847530176s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.95761565s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:13:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.957638245s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.882743253s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:12:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.882782319s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 3.052066734s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:12:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.052096497s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.988883532s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:12:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.988917569s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)