Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.015044213s trace: https://jaeger.dev.storj.io/trace/58ac67eeb4b72dad FAIL (2025-12-17 01:44:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.01507523s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.095170801s trace: https://jaeger.dev.storj.io/trace/58ac67eeb4b72dad FAIL (2025-12-17 01:43:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.095259973s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.034844143s trace: https://jaeger.dev.storj.io/trace/58ac67eeb4b72dad FAIL (2025-12-17 01:42:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.034869716s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002155851s trace: https://jaeger.dev.storj.io/trace/58ac67eeb4b72dad FAIL (2025-12-17 01:42:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002180807s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.408612794s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:11:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.408647033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.383145229s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:11:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.383180599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.374158554s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:09:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.374184478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.356215056s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:09:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.356242793s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.508145906s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:08:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.50817669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.33373544s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:07:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.333761627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.515346577s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:06:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.515375611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.600476884s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:06:03 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.600521851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.754090222s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:05:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.754178504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.397442918s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:04:03 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.397474769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.468688301s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:03:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.468725234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.370078811s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:03:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.370105698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.371313647s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:02:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.371344573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.429162986s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:01:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.429198058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.520346476s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:00:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.52038487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.355115482s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 22:00:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.3552159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.40913922s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:59:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.40916646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.336807422s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:58:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.336830031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.437833084s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:58:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.437982814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.323333913s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:57:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.323368672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.330385653s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:56:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.330415257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.465013086s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:56:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.465044941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.446984242s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:55:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.447017147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.441423471s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:54:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.441454795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.372680386s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:53:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.372707865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.256227452s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:52:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.256268194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.329943216s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:51:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.329976558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.543538602s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:51:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.543571539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.47977592s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:50:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.479807677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.423620936s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:49:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.423647548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.292528308s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:48:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.292723822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.355286326s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:47:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.355314915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.270216481s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:46:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.270241461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.439411147s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:46:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.43943769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.323982603s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:45:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.324011866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.334496521s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:44:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.334522869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.377992132s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:43:51 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.378076086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.386745928s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:43:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.386778472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.284892559s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:42:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.284997922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.272312508s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:42:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.272341241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.388232363s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:41:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.3882647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.340211867s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:40:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.34025142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.391726791s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:39:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.391763166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.286836248s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:38:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.286872669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.389093732s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:37:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.389132266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.386968512s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:36:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.387036166s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)