Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 58.068629ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6408175d89e1a801 FAIL (2025-12-18 00:02:59 UTC) orbiter warning: Get "https://www.storj.io/": EOF Duration: 58.09704ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 54.548474ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6408175d89e1a801 FAIL (2025-12-18 00:01:49 UTC) orbiter warning: Get "https://www.storj.io/": EOF Duration: 54.574274ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 23.92998ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6408175d89e1a801 FAIL (2025-12-18 00:00:35 UTC) orbiter warning: Get "https://www.storj.io/": EOF Duration: 24.033245ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.435016301s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:20:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.435042863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.872572141s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:19:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.872599935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.510326599s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.51035656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.474766611s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.474791764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.363837342s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:17:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.363928196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.42096755s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.42101003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.403797857s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.403888033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 2.416904419s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:16:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.41693213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.550141305s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.550170618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.883367723s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:15:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.883503349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.357854775s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.357883591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.244682409s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:15:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.244710801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.411527423s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.411600528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.243221581s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:14:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.243247761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.415183077s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.415212368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.252852197s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:13:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.252870003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.394881134s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:13:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.39490713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.358729939s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:12:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.358762357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.335201733s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:12:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.335231551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.382565906s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:12:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.382580121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.564786443s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:11:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.564875933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.487804415s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:11:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.487845587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.249268905s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:11:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.249286027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.491291039s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:10:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.491322325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 2.429632193s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:10:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.429657242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.653629541s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:10:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.653658223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 2.43180357s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.431830955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.427694713s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:09:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.42780863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 2.397709105s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:08:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.397735718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.395350096s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:08:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.395411698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 2.384746297s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:07:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.384769053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.453713423s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:07:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.453747207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.309003294s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:07:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.309031659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 2.465295941s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:06:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.465326922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 2.364179337s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:06:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.364204382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.406728126s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:06:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.406757739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 2.753281503s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:05:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.753475489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.282535638s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:05:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.282565976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 2.39466379s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:05:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.394683211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 2.34971835s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:04:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.349816627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.510754673s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.510780139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.401513161s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:04:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.40154211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.243340819s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:03:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.243385462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 2.384906423s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:03:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.385002963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.303370214s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:03:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.303451035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 2.401715446s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:03:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.401746265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.844024434s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:02:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.844057139s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)