Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.364572604s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:44:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.364598527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 2.480785295s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:43:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.480815202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.753973375s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:43:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.754002931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 2.393820014s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.393911369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.29848108s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:42:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.298516442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 2.428555186s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:42:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.428584106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.447656074s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:41:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.447681894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 2.366031298s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:41:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.366058229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.547413772s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:40:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.547439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 2.431264317s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:40:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.431289597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 2.28159148s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.281692696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.434612514s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:40:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.434642387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 2.5289407s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.529040097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.305944698s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:39:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.305975905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 2.533447024s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:38:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.53347499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.447277067s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:38:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.447304339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.495277163s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:38:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.495307179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 2.295432422s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:37:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.295463146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.53455124s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:37:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.534578776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 2.38698596s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:37:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.38709464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.61878216s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:37:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.618819161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.003499188s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:36:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 30.003516008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.005118684s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:36:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 1m0.005138409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003941502s trace: https://jaeger.dev.storj.io/trace/2e2a7c574013d338 FAIL (2025-12-17 18:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 35:59.928876933 (291h7m39.909229561s) [upload=256049]: Uploading segments... Dec 17 18:35:59.929104019 (291h7m39.909456646s) [upload=256049]: Got next segment. Inline: false Dec 17 18:35:59.929148554 (291h7m39.909501202s) [upload=256049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:35:59.929156988 (291h7m39.909509622s) [upload=256049]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:00.055643999 (291h7m40.035996646s) [seg_pos={0 0}, upload=256049]: Begin upload segment... Dec 17 18:36:00.055710625 (291h7m40.036063262s) [seg_pos={0 0}, upload=256049]: Done begin upload segment. Dec 17 18:36:00.055743978 (291h7m40.036096622s) [noise=true , address=145.79.227.45:4231 , piece_id=7XLGW3ZG726GHXL7VCIHIQX65UJJYJAL22MTXT4YY6YTRGIPMFHQ, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.063876567 (291h7m40.044229211s) [noise=true , address=145.79.227.51:4411 , piece_id=ZADBT3UTMXY5Y3LR3YGYLG7Y2OIZ6ZV7NGATRW7EBDPVX6K5DA5Q, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.066303847 (291h7m40.046656489s) [noise=true , address=145.79.227.27:4690 , piece_id=PTZJGKYU7FDUMP4LWT4S7QVRX7PIKGNMTKI46QPFN7H3SZFR437Q, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.069339647 (291h7m40.049692294s) [noise=true , address=145.79.227.35:4483 , piece_id=YLUD54XQO7NK7JJZTCRJPYILJ54LKLI7LLFYID6NPJ6AQDLNITBA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.071678598 (291h7m40.052031245s) [noise=true , address=45.91.116.14:4190 , piece_id=MZKRFUVAETIIW2MR7JRO4HIC4NL3FPPURCKWADJ6BJDIB36GW25A, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.075014183 (291h7m40.055366838s) [noise=true , address=145.79.227.32:4835 , piece_id=EIKNF62YLH67XZDAQ4GZQ5TABSPXPA3QV6MA2YS7FEBJGJWT3PGA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.081101022 (291h7m40.061453668s) [noise=true , address=45.91.116.7:4011 , piece_id=TW5QYSWCJWCXUGKOVOWCPGM7ZRZJDQNULZDCYLLKJKKSFRKC7BFA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.085440834 (291h7m40.06579348s) [noise=true , address=145.79.227.31:4797 , piece_id=YRWPBEDJRSG4QUVOIOA6BV3IMMKPRPMZC7B7LIBUCTOP533772JA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.087741538 (291h7m40.06809418s) [noise=true , address=145.79.227.48:4302 , piece_id=LBXHSZZAZDF3VHQYFCKTHYYL2QVYVAWYD7QEME4ANG5NVZ3727LQ, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.090578685 (291h7m40.070931327s) [noise=true , address=145.79.227.49:4339 , piece_id=O7AKVZXRWSRIPAUW4BTRD7CXLNEX75UKCGMXZP3S5KTUELAZ4UVA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.097881089 (291h7m40.078233728s) [noise=true , address=145.79.227.37:4566 , piece_id=Q2VQLI7CCGNZFTXGDPM54APVNPJZIDPS62XEOQ72YGT3JXKOHVXQ, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.098619412 (291h7m40.078972058s) [noise=true , address=77.90.31.13:4627 , piece_id=I2YI62EBQGHR2QAI2MNIWCNCCUHPLYGY3EIVR3KRJIP5XCOROQBA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.100098826 (291h7m40.080451494s) [noise=true , address=145.79.227.46:4601 , piece_id=23NLEVE46G3WB2EFTEFERH3CXEL676XUMMGTNYT3ZKDZUGO7SKNA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.102153940 (291h7m40.08250659s) [noise=true , address=145.79.227.39:4013 , piece_id=TYW4AP7K2TSSGVZ2ABS6Z2CDFK2HS6VI6X3QN4P7YGSO2J3HXE3Q, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.104358133 (291h7m40.084710779s) [noise=true , address=145.79.227.22:4511 , piece_id=AC56C5UL2XJ57THHI6W4DV4F55HCLBBJATD43OYN3S2R7NCR2PYA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.104958152 (291h7m40.085310828s) [noise=true , address=145.79.227.28:4725 , piece_id=7IHFGQS4NQUMKSQM2BRU556EKBP223VZ53WWIEW2JZGTC6MDKPTA, seg_pos={0 0}, upload=256049]: Uploading piece... Dec 17 18:36:00.864378902 (291h7m40.84473155s) [upload=256049]: Next returned nil segment Dec 17 18:36:00.864396949 (291h7m40.844749597s) [upload=256049]: Waiting for error group managing segments... Dec 17 18:36:00.987709929 (291h7m40.968062673s) [noise=true , address=145.79.227.37:4566 , piece_id=Q2VQLI7CCGNZFTXGDPM54APVNPJZIDPS62XEOQ72YGT3JXKOHVXQ, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:01.086967104 (291h7m41.06731974s) [noise=true , address=77.90.31.13:4627 , piece_id=I2YI62EBQGHR2QAI2MNIWCNCCUHPLYGY3EIVR3KRJIP5XCOROQBA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:01.198894471 (291h7m41.179247112s) [noise=true , address=45.91.116.14:4190 , piece_id=MZKRFUVAETIIW2MR7JRO4HIC4NL3FPPURCKWADJ6BJDIB36GW25A, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:02.573242473 (291h7m42.553595117s) [noise=true , address=45.91.116.7:4011 , piece_id=TW5QYSWCJWCXUGKOVOWCPGM7ZRZJDQNULZDCYLLKJKKSFRKC7BFA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:03.669725744 (291h7m43.65007839s) [noise=true , address=145.79.227.28:4725 , piece_id=7IHFGQS4NQUMKSQM2BRU556EKBP223VZ53WWIEW2JZGTC6MDKPTA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:04.813771911 (291h7m44.794124548s) [noise=true , address=145.79.227.39:4013 , piece_id=TYW4AP7K2TSSGVZ2ABS6Z2CDFK2HS6VI6X3QN4P7YGSO2J3HXE3Q, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:05.247108952 (291h7m45.227461592s) [noise=true , address=145.79.227.48:4302 , piece_id=LBXHSZZAZDF3VHQYFCKTHYYL2QVYVAWYD7QEME4ANG5NVZ3727LQ, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:05.880832347 (291h7m45.861184992s) [noise=true , address=145.79.227.27:4690 , piece_id=PTZJGKYU7FDUMP4LWT4S7QVRX7PIKGNMTKI46QPFN7H3SZFR437Q, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:06.393338543 (291h7m46.373691194s) [noise=true , address=145.79.227.31:4797 , piece_id=YRWPBEDJRSG4QUVOIOA6BV3IMMKPRPMZC7B7LIBUCTOP533772JA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:06.732777263 (291h7m46.713129906s) [noise=true , address=145.79.227.32:4835 , piece_id=EIKNF62YLH67XZDAQ4GZQ5TABSPXPA3QV6MA2YS7FEBJGJWT3PGA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:10.248211159 (291h7m50.228563795s) [noise=true , address=145.79.227.46:4601 , piece_id=23NLEVE46G3WB2EFTEFERH3CXEL676XUMMGTNYT3ZKDZUGO7SKNA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:15.216567704 (291h7m55.196920332s) [noise=true , address=145.79.227.35:4483 , piece_id=YLUD54XQO7NK7JJZTCRJPYILJ54LKLI7LLFYID6NPJ6AQDLNITBA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:28.971650508 (291h8m8.952003145s) [noise=true , address=145.79.227.22:4511 , piece_id=AC56C5UL2XJ57THHI6W4DV4F55HCLBBJATD43OYN3S2R7NCR2PYA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: <nil> Dec 17 18:36:28.971691214 (291h8m8.95204384s) [seg_pos={0 0}, upload=256049]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:28.972256777 (291h8m8.952609418s) [noise=true , address=145.79.227.51:4411 , piece_id=ZADBT3UTMXY5Y3LR3YGYLG7Y2OIZ6ZV7NGATRW7EBDPVX6K5DA5Q, seg_pos={0 0}, upload=256049]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 139.59.144.22:59756->145.79.227.51:4411: use of closed network connection; write tcp 139.59.144.22:59756->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59756->145.79.227.51:4411: use of closed network connection Dec 17 18:36:28.972273628 (291h8m8.952626271s) [noise=true , address=145.79.227.51:4411 , piece_id=ZADBT3UTMXY5Y3LR3YGYLG7Y2OIZ6ZV7NGATRW7EBDPVX6K5DA5Q, seg_pos={0 0}, upload=256049]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.972755110 (291h8m8.953107757s) [noise=true , address=145.79.227.45:4231 , piece_id=7XLGW3ZG726GHXL7VCIHIQX65UJJYJAL22MTXT4YY6YTRGIPMFHQ, seg_pos={0 0}, upload=256049]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: write tcp 139.59.144.22:37896->145.79.227.45:4231: use of closed network connection; write tcp 139.59.144.22:37896->145.79.227.45:4231: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37896->145.79.227.45:4231: use of closed network connection Dec 17 18:36:28.972767344 (291h8m8.953119992s) [noise=true , address=145.79.227.45:4231 , piece_id=7XLGW3ZG726GHXL7VCIHIQX65UJJYJAL22MTXT4YY6YTRGIPMFHQ, seg_pos={0 0}, upload=256049]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.973267222 (291h8m8.95361986s) [noise=true , address=145.79.227.49:4339 , piece_id=O7AKVZXRWSRIPAUW4BTRD7CXLNEX75UKCGMXZP3S5KTUELAZ4UVA, seg_pos={0 0}, upload=256049]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 17 18:36:28.973281020 (291h8m8.953633679s) [noise=true , address=145.79.227.49:4339 , piece_id=O7AKVZXRWSRIPAUW4BTRD7CXLNEX75UKCGMXZP3S5KTUELAZ4UVA, seg_pos={0 0}, upload=256049]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.973311282 (291h8m8.953663919s) [seg_pos={0 0}, upload=256049]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:28.973635300 (291h8m8.953987936s) [upload=256049]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.973642103 (291h8m8.953994742s) [upload=256049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:29.119933274 (291h8m9.100285916s) [upload=256049]: Done uploading segments... Dec 17 18:36:29.119953413 (291h8m9.100306049s) [upload=256049]: Upload finished. err: <nil> Duration: 30.003964451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 992.907024ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:31:12 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 992.952132ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 933.456351ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:30:36 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 933.545907ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 886.511685ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:30:17 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 886.541251ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 966.095288ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:30:02 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 966.190376ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 836.844404ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:29:26 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 836.872017ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 956.280746ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:28:56 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 956.309714ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 916.415638ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:28:35 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 916.441933ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 915.981751ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:28:02 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 916.005655ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 848.251849ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:27:42 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 848.278434ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 878.057986ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:26:45 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 878.084244ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 888.904079ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:25:42 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 888.939644ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 904.725118ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:25:09 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 904.763075ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 902.158281ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:24:13 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 902.187531ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 948.728949ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:23:34 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 948.756658ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 890.853618ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:23:11 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 890.882639ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 944.949746ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:21:59 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 944.974512ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 978.0487ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:21:01 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 978.071022ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 932.164599ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:20:22 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 932.255857ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 903.865664ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:19:51 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 903.903491ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 913.295872ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:19:35 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 913.327158ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 939.862242ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:19:00 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 939.902911ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.017806155s trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:18:05 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.017837899s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 932.88115ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:17:29 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 932.898214ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 952.742507ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:17:00 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 952.775733ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 946.708424ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:16:30 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 946.799869ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 925.08691ms trace: https://jaeger.dev.storj.io/trace/30964f87c8c8c404 FAIL (2025-12-17 07:15:48 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 925.113817ms core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)