Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.145246893s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:04:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.145342868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.134723116s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:03:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.134752136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.949773269s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.949822224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 3.637799215s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:01:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.637823087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.170804276s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.17083569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.022039173s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:59:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.022602133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.085238363s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:58:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.085266432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.059059765s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:57:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.059086558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.029785094s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:56:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.029811428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.108945338s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:55:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.108971016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.919667355s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:54:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.919689917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.12011518s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:54:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.12014415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.193118166s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.193143803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.121940722s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.121974975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.016965378s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:51:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.016989149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 3.149806663s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:51:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.149853804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.114783131s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 14:51:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.114803031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.618688015s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:44:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.618732438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.606566741s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:43:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.606610489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.842383291s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:43:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.842415788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.653861004s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:42:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.653886703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.532874608s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:42:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.532901203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.662180215s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:41:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.662217049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.566367856s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:40:39 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.566387563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.553633564s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:40:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.553736263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.604855229s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:39:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.604896777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.562194494s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:38:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.562283964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.588701702s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:37:12 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.588732368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.551361877s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:36:47 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.551388812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.700243207s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:36:29 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.700271184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.568060479s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:36:12 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.56808429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.583068183s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:35:47 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.583090978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.64560949s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:35:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.645629363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 2.116344317s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:34:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.11637925s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.628791949s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:33:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.628814279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.544506136s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:33:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.544536552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.605280441s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:32:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.605316505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.593729107s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:31:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.593759341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.650167491s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:30:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.650209834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.604915785s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:30:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.604942426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 1.578521247s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:29:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.578545501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.546653809s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:29:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.546686586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.58419415s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:28:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.584226537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.608758722s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:27:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.60879237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.605368792s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:26:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.605396398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.583602146s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:25:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.583698535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 1.624951299s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:24:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.624991521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.610124957s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:23:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.610153489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 1.603337877s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:22:54 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.603374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.621759912s trace: https://jaeger.dev.storj.io/trace/330ce0f3d3fb12f8 FAIL (2026-03-07 10:21:59 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.621809549s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)