Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.013156622s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:28:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.013185361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.941952761s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.942040445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.148573204s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:27:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.14869025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.062524724s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:26:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.062548277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.171548531s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:25:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.17157784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.675321885s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:24:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.675345281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.056651685s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.056679059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.999715951s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:23:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.999739626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 3.190977309s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:22:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.191014381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.032686407s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:21:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.032712392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.172761839s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:21:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.172808071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 3.030801412s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:21:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.030828481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.021879577s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:20:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.021913359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.242731998s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:19:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.242758894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.129070256s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.129092093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.935874377s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.935901901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.950370482s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:17:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.950416401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.981953464s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:16:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.982085362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.22089936s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:15:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.220924823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.212643429s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:14:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.212685664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.054798181s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:13:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.054825188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.167278771s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:12:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.167309103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.083164166s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:12:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.08319375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.049655863s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.049685736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.921697821s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:11:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.921734185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.019254594s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.019285424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.071558979s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:09:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.071588055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.772444696s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.772473116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 3.760854373s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:08:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.760884466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 3.096201941s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:07:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.096293558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 3.129122581s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:06:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.12915269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 35.046063617s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:06:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.046089978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 3.235976693s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:04:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.235996384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.145246893s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:04:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.145342868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.134723116s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:03:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.134752136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.949773269s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.949822224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 3.637799215s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:01:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.637823087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.170804276s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.17083569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.022039173s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:59:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.022602133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.085238363s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:58:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.085266432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.059059765s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:57:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.059086558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.029785094s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:56:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.029811428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.108945338s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:55:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.108971016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.919667355s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:54:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.919689917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.12011518s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:54:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.12014415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.193118166s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.193143803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.121940722s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.121974975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.016965378s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:51:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.016989149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 3.149806663s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:51:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.149853804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.114783131s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:51:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.114803031s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)