Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 2.416573707s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.416602079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.306855753s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:01:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.306884138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 2.413280335s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:01:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.413316882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 2.391480714s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 19:00:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.391524573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.380946608s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 19:00:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.380970407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.501362241s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:59:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.501394923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 2.469443264s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:59:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.46947497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 2.426825115s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:58:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.426850308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.605790516s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:58:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.605839589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.670470769s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:57:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.670493874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 2.323854319s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:57:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.323881237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 2.396044638s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.396162522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 2.336232824s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:56:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.336257617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.225028178s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:56:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.22505329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 2.345589286s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:56:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.345619226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.55431737s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:56:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.554410969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 2.398108349s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.398144285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 2.443053018s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:55:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.443085013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.504558595s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:55:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.50458383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 2.459612277s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:54:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.45964176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.418907516s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:54:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.418939957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 2.424046755s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.424076686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.395116388s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:53:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.39522994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 2.395966022s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:52:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.39599411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.251835261s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:52:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.251864767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 2.308680692s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:52:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.308703229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 2.408514006s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.408562893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.650648842s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.65067759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.333868712s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:51:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.333963004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 2.39064945s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:51:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.390692234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 2.440141634s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:50:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.440167751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.917092174s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:50:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.917126701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.330796926s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:50:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.330823742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 2.312920448s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:49:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.312955424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.263849887s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:49:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.26386743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 2.495468578s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.495497859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.412721482s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:48:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.412748932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 2.279316277s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.279398188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.321704382s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:48:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.321731662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 2.305364334s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:47:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.305392565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.317611269s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:47:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.317643116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.375861408s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:47:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.375894714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 2.496138307s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.496164795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 2.416580018s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.41660524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.367268717s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:45:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.367316047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 2.372075283s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:45:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.372101661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.81894551s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:45:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.818976509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.420822997s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:45:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.420851474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.373045824s trace: https://jaeger.dev.storj.io/trace/76f1536543f407f5 FAIL (2025-12-17 18:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :36:00.048430653 (291h7m40.02878331s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049111099 (291h7m40.029463745s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049255820 (291h7m40.029608459s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.049826321 (291h7m40.030178969s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050156989 (291h7m40.030509629s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.050661159 (291h7m40.031013799s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051195324 (291h7m40.031547963s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.051512675 (291h7m40.031865318s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052091941 (291h7m40.032444583s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052568099 (291h7m40.032920723s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.052794030 (291h7m40.033146672s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.053493751 (291h7m40.033846388s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.058279304 (291h7m40.038631999s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.094093837 (291h7m40.074446469s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Uploading piece... Dec 17 18:36:00.713542651 (291h7m40.693895291s) [upload=256048]: Next returned nil segment Dec 17 18:36:00.713560272 (291h7m40.693912913s) [upload=256048]: Waiting for error group managing segments... Dec 17 18:36:00.894747565 (291h7m40.875100203s) [noise=true , address=5.231.73.5:4019 , piece_id=TCJAEU47ENIXGNJFPSTCLHRYYX42SFROPI7CNB3USMG2IOQNWEUA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.902222885 (291h7m40.882575524s) [noise=true , address=5.231.73.4:4006 , piece_id=NCXV4JB5UF3YSCN57RV2E4UIUWH4FZHCEY4HLCRGDP4FGWUHQH5A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:00.926404791 (291h7m40.906757427s) [noise=true , address=5.231.73.3:4020 , piece_id=YHQSZVYUKM2VAEB4WU2K5VKNUWJK6IGHIVPWA2WTYGU5ZK33FY3A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.000336277 (291h7m40.980688952s) [noise=true , address=77.90.31.15:4727 , piece_id=YNWG5TCZAAH33LR5MVHPZCAQU6Z4UEMKLBNVBAOY2F7XNZTWWBKA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.254858413 (291h7m41.235211055s) [noise=true , address=145.79.227.34:4915 , piece_id=HHN77Z3QHJKDTECXJJJRKZ75MJ2D75EJM42AGHMMR6YEMK4ASIOQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.349459040 (291h7m41.329811679s) [noise=true , address=77.90.31.13:4648 , piece_id=FV4CS4J2BDWGD6LKH7KA5BJRLQFV5EIEBUE7WNHTTF2JV5YG7IPA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:01.790065128 (291h7m41.77041778s) [noise=true , address=77.90.31.16:4766 , piece_id=7TZZFBPI5TCFWAHBQAR3NYMRF5XE365WFN4XT4FG3SYY6ZUCLI4A, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:04.318202050 (291h7m44.29855469s) [noise=true , address=45.91.116.6:4001 , piece_id=2UIYMHHXGJ3EBP7TPIJDVAF7RCMUOVI3MEHHZ7D6WHUWL3BSAEWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.183175619 (291h7m45.163528262s) [noise=true , address=64.49.13.27:4965 , piece_id=WAKQ4O2P3PL6CFVIOKG5NV3PSZ6ILUONYIITG6XCBOEPA3OBVKLA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:05.996753731 (291h7m45.97710638s) [noise=true , address=77.90.31.12:4605 , piece_id=ZFTQFOMIYQVSZKRTDTE77VQYWAV3ITFJEVQDFENLESVM7JDLK3TQ, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.040183185 (291h7m48.020535825s) [noise=true , address=145.79.227.44:4209 , piece_id=HP4UO24PY2NS4YQIS6H2HACSONQK7GNWFF6HR6VH3VAY4MO6I75Q, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:08.671124012 (291h7m48.651476657s) [noise=true , address=145.79.227.38:4984 , piece_id=3QJ2RHSS4ERNZWMDE2TEG4ZOKLCH53GCTM75FGPBQIQA2LZ2K4RA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:25.801401249 (291h8m5.781753886s) [noise=true , address=145.79.227.13:4806 , piece_id=EWUVMQZEIO5JW7L2IABSZ3HKR2YBNCO636QNYCUVUDNOG2NVQLDA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772412359 (291h8m8.752764998s) [noise=true , address=145.79.227.14:4858 , piece_id=PCMT4DMUIGFNIBAJS44ERSCRTAAVDQSX27BT3UL7C7KAMRLYZORA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: <nil> Dec 17 18:36:28.772460943 (291h8m8.75281357s) [seg_pos={0 0}, upload=256048]: Segment reached optimal threshold of 14 pieces. Dec 17 18:36:28.772850275 (291h8m8.753202917s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: read tcp 139.59.144.22:55560->64.49.13.169:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.772862902 (291h8m8.753215559s) [noise=true , address=64.49.13.169:4708 , piece_id=7HZQTEW7QV2522QC3CNQ4RWI5WY6J3JLUFRXO44UVZLVLBB6TEYA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.772989524 (291h8m8.753342169s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60362->45.91.116.12:4238: use of closed network connection Dec 17 18:36:28.773008492 (291h8m8.753361134s) [noise=true , address=45.91.116.12:4238 , piece_id=TUQW3KEAZVEMI7EIULPAJ6UZLBFJ7B4U7HZ3GRAC2N4PDRHO4DWA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773045381 (291h8m8.753398026s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35000->77.90.31.11:4564: use of closed network connection Dec 17 18:36:28.773059427 (291h8m8.753412069s) [noise=true , address=77.90.31.11:4564 , piece_id=3A3TQVMTBX6FRGULLYZGFALWOPNXYUAEKROMEZFSLPH4SBD7CRGA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773269262 (291h8m8.753621908s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bFAUM5PA2gqHYVo7gQhKb5RHoVFxKFf44xoZhXhMuT7jYJi7b): protocol: read tcp 139.59.144.22:40848->64.49.13.139:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:28.773279751 (291h8m8.753632393s) [noise=true , address=64.49.13.139:4469 , piece_id=HAIHN772F3MWP7442OFQGOSHI3CC5FYCNXA4QRQVX65WBJ7MMNIA, seg_pos={0 0}, upload=256048]: Long tail cancellation due to optimal threshold Dec 17 18:36:28.773304407 (291h8m8.753657032s) [seg_pos={0 0}, upload=256048]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 17 18:36:28.773417554 (291h8m8.753770191s) [upload=256048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:28.773425709 (291h8m8.753778362s) [upload=256048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:28.913108886 (291h8m8.893461527s) [upload=256048]: Done uploading segments... Dec 17 18:36:28.913133348 (291h8m8.893485989s) [upload=256048]: Upload finished. err: <nil> Duration: 2.373077631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 2.388572141s trace: https://jaeger.dev.storj.io/trace/2a43eaa5fab79951 FAIL (2025-12-17 18:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA) context deadline exceeded ======LOGS====== ad=256056]: Done starting upload Dec 17 18:36:30.061810899 (291h8m10.042163545s) [upload=256056]: Uploading segments... Dec 17 18:36:30.062058510 (291h8m10.042411158s) [upload=256056]: Got next segment. Inline: false Dec 17 18:36:30.062189215 (291h8m10.042541844s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 18:36:30.062221120 (291h8m10.042573771s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 18:36:30.184607878 (291h8m10.164960518s) [seg_pos={0 0}, upload=256056]: Begin upload segment... Dec 17 18:36:30.184691839 (291h8m10.165044487s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.184755306 (291h8m10.16510795s) [seg_pos={0 0}, upload=256056]: Done begin upload segment. Dec 17 18:36:30.184786551 (291h8m10.165139195s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.185223296 (291h8m10.165575947s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.193155962 (291h8m10.173508613s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.194410093 (291h8m10.174762744s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.195504689 (291h8m10.175857341s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.196697937 (291h8m10.177050587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.197838986 (291h8m10.178191638s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.198744023 (291h8m10.179096719s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.200464822 (291h8m10.180817463s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.201316596 (291h8m10.181669247s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.202381490 (291h8m10.182734136s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.203258245 (291h8m10.183610895s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.204440080 (291h8m10.184792728s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.205253906 (291h8m10.185606553s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.206381613 (291h8m10.186734266s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Uploading piece... Dec 17 18:36:30.834054012 (291h8m10.814406663s) [upload=256056]: Next returned nil segment Dec 17 18:36:30.834099614 (291h8m10.814452261s) [upload=256056]: Waiting for error group managing segments... Dec 17 18:36:30.972069408 (291h8m10.95242205s) [noise=true , address=77.90.31.13:4627 , piece_id=BRQKBS3M5G4F4Y45FNUETIL4S5ZG7WYYECARHUE43HW4PCHTBIUA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:30.981334358 (291h8m10.961687005s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTYWA43AOI75HYQPAMYA4DRMWL4XIQSWE23UHDTCLJSRBBWWBBSA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:31.522688984 (291h8m11.50304162s) [noise=true , address=145.79.227.38:4977 , piece_id=6WSU6OBJXCWI6ZATUP6MSJD43T63QCILJQ4C7YEFSYIDH3K77MIA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.068683905 (291h8m12.049036552s) [noise=true , address=45.91.116.13:4154 , piece_id=LB655SGL2SKDRF24YG3J4KI2S3UAVFWENNZVQUTOOQ4H5TF6AW6Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:32.577860160 (291h8m12.558212796s) [noise=true , address=145.79.227.46:4602 , piece_id=3FIZEUHKJSHPE3GQCYPR3K6KP2MZCSJQKQ7SRNS7564TUS7CZ6FQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:33.083162592 (291h8m13.063515238s) [noise=true , address=145.79.227.37:4567 , piece_id=I2BFD22JKCKSJ7P2VPBTZF6P3ZA56MR2KHTDM462VJPGCUCIGXAA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:35.646075304 (291h8m15.626427941s) [noise=true , address=145.79.227.43:4159 , piece_id=NKBGRXIQVDNVJK7RQZRL5D5CPZOQEGBNOMNKPQLOEDME7BQFQY2Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.299934428 (291h8m19.280287069s) [noise=true , address=145.79.227.48:4303 , piece_id=W5E2TL5AIJPBQWSO5D572ZQ2S73VFJNRQTRHX43SWVTNDHYATO3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:39.632235564 (291h8m19.612588203s) [noise=true , address=145.79.227.50:4374 , piece_id=RDOWYVEKEGU7RVEKMLMH2D4MNYMZOVYDEXUS5XEOEDNEO3S5PXVA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:45.636407032 (291h8m25.616759669s) [noise=true , address=145.79.227.32:4836 , piece_id=I3VZH3CK3MBNP7EK7YZW2VLHS6UIJ3QIYE37TV5PRUIRJVOK5NAQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:47.567794131 (291h8m27.548146769s) [noise=true , address=145.79.227.21:4474 , piece_id=6KHMPOOE7SGK7ABNX2Q7C3JGUX5YHRM7RDN6NDVY5OW3DSWHNNBQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:49.724821639 (291h8m29.705174279s) [noise=true , address=145.79.227.35:4483 , piece_id=NX4I6H6NWYPRK4WUVWDENMRSYHU5UGDJJX3HB37MZSEYHMKMHRNQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290484199 (291h8m38.270836835s) [noise=true , address=145.79.227.47:4265 , piece_id=2RB6SVSCQSWRRYS4AZWXQASY7KLAYHLCTTUO5DZ6RNBKOHILZIWA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: <nil> Dec 17 18:36:58.290501985 (291h8m38.270854624s) [seg_pos={0 0}, upload=256056]: Segment reached optimal threshold of 13 pieces. Dec 17 18:36:58.290788788 (291h8m38.271141449s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54226->145.79.227.52:4447: use of closed network connection Dec 17 18:36:58.290805735 (291h8m38.271158378s) [noise=true , address=145.79.227.52:4447 , piece_id=YUKZDQZOKUVQDHWI5CXTJKSLMZEAXETEC5M656764KHQZCSXXNMQ, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290848976 (291h8m38.271201618s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: rpc: dial tcp 145.79.227.36:4517: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.290856514 (291h8m38.271209153s) [noise=true , address=145.79.227.36:4517 , piece_id=AL7E32DRM63JYI6MWPPIGLD65BQSCOOTWQPELJUPSDVMZ5BIQQMA, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.290993590 (291h8m38.271346226s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: read tcp 139.59.144.22:42464->145.79.227.19:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 17 18:36:58.291000569 (291h8m38.271353204s) [noise=true , address=145.79.227.19:4077 , piece_id=ZZCOVAKTDQIY5OBV5BDPVLY4NCPSDYPJ7OBUKHXETXRVBM6EZK3Q, seg_pos={0 0}, upload=256056]: Long tail cancellation due to optimal threshold Dec 17 18:36:58.291060706 (291h8m38.271413329s) [seg_pos={0 0}, upload=256056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 18:36:58.291132244 (291h8m38.27148488s) [upload=256056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 18:36:58.291136884 (291h8m38.27148952s) [upload=256056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 18:36:58.433558224 (291h8m38.41391087s) [upload=256056]: Done uploading segments... Dec 17 18:36:58.433581424 (291h8m38.413934064s) [upload=256056]: Upload finished. err: <nil> Duration: 2.388597883s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)