Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.491701352s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:12:02 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.491732754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.453096806s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:11:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.453126213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.428504494s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:10:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.428538915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.464224755s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:10:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.464252007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.333396226s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:09:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.333424538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.515978623s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:09:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.516009604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.426825254s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:08:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.42685138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.49565851s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:07:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.495688094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.483082742s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:07:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.483172873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.714146086s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:06:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.714178508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.38723612s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:05:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.387262648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.37453736s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:05:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.374564887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.468638739s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:04:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.468668125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.344805091s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:03:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.344959331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.51753198s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:02:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.517565403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.385735621s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:01:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.385759574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.506795775s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:01:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.506824811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.377337089s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:00:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.377368149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.37871471s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:59:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.378747822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.430154792s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:58:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.430179148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.466001338s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:57:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.466061143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.501965438s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:57:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.50215122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.393443699s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:56:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.393472738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.338130488s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:55:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.338162562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.435906299s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:55:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.43593438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.425756195s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:54:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.425834242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.368485941s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:54:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.368514279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.490381289s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:53:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.490405726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.448579281s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:52:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.448605764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.430953612s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:52:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.431057638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.487329511s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:51:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.487361786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.495411242s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:50:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.495440417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.4249999s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:49:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.425033166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.38660471s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:48:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.386630612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.416838006s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:48:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.416865363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.378691815s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:47:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.378719093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.42356749s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:47:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.42358983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.353203909s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:46:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.353247149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.41451119s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:45:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.414544105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.354391501s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:44:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.35452248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.467401672s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:44:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.467430456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.489943382s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:43:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.489965491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.411955207s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:43:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.41198351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.381731972s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:42:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.38175917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.49450265s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:41:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.49469786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003536717s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 08:40:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 30.003564081s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.778664712s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:42:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.77869045s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.944891572s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.944926151s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.782781999s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:41:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.782810233s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.77282873s trace: https://jaeger.dev.storj.io/trace/42206651daeeda15 FAIL (2025-11-14 03:40:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.772922785s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)