Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.562045482s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:36:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.562273107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.342580726s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:35:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.342608779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.329074498s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:34:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.329103668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.469425185s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:34:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.469455012s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.31993418s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:33:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.319964138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.492069351s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:32:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.492093583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.405690802s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:31:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.405722507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.337872185s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:31:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.337914298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.311762568s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:30:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.311793149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.647878317s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:30:02 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.647972169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.481806111s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:29:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.481843213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.473141207s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:28:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.473167928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.459004676s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:28:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 2.459032208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00565805s trace: https://jaeger.dev.storj.io/trace/1ba48bd573036878 FAIL (2025-12-16 21:28:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 21:27:32.258979559 (269h59m12.239332199s) [upload=237383]: Starting upload Dec 16 21:27:32.259061593 (269h59m12.23941423s) [upload=237383]: Done starting upload Dec 16 21:27:32.259267285 (269h59m12.239619921s) [upload=237383]: Uploading segments... Dec 16 21:27:32.259666152 (269h59m12.240018788s) [upload=237383]: Got next segment. Inline: false Dec 16 21:27:32.259720640 (269h59m12.240073276s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 21:27:32.259727296 (269h59m12.240079932s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 21:27:32.375034947 (269h59m12.355387589s) [seg_pos={0 0}, upload=237383]: Begin upload segment... Dec 16 21:27:32.375099879 (269h59m12.35545252s) [seg_pos={0 0}, upload=237383]: Done begin upload segment. Dec 16 21:27:32.375258950 (269h59m12.355611592s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.375333304 (269h59m12.355685951s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376033145 (269h59m12.356385794s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.376675917 (269h59m12.357028564s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.377309998 (269h59m12.357662647s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378030685 (269h59m12.358383333s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378510617 (269h59m12.358863264s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.378812102 (269h59m12.359164751s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379447378 (269h59m12.359800018s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379587808 (269h59m12.359940456s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.379995962 (269h59m12.360348603s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380106761 (269h59m12.360459407s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380530350 (269h59m12.360882991s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.380566889 (269h59m12.360919538s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381018036 (269h59m12.361370682s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.381471709 (269h59m12.361824356s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Uploading piece... Dec 16 21:27:32.991817536 (269h59m12.97217018s) [upload=237383]: Next returned nil segment Dec 16 21:27:32.991834862 (269h59m12.972187506s) [upload=237383]: Waiting for error group managing segments... Dec 16 21:27:33.133548710 (269h59m13.113901359s) [noise=true , address=45.91.116.13:4156 , piece_id=MGSS35STI2RQO2IVBTJDP74H6YTDWHQ4K6RZZK2JBSR6BBEM4E2Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.140489191 (269h59m13.120841831s) [noise=true , address=45.91.116.12:4226 , piece_id=YJJ66OZSHRPJUWGUTJDXRXNWKZIX3V4CHRRJYKNEWBNNCAAXB7OA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.161225409 (269h59m13.141578046s) [noise=true , address=45.91.116.5:4297 , piece_id=Q35QTNZDYL6OAEHH47L2GBV3JEJBRYXST654CE6FYAHFOJF6VTGQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.215372421 (269h59m13.195725057s) [noise=true , address=45.91.116.4:4937 , piece_id=GZXY6O3EHFTDZATIRA4LYA4U3PDY7MXV3WM3SWW72AJW5ACEM4OQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.219204977 (269h59m13.199557613s) [noise=true , address=145.79.227.15:4875 , piece_id=WPWZ3HPY3YJ36JCNWDXO3KZSZ2JHJLDVW56ZJSIJEFTM4LV2PD3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.234696826 (269h59m13.215049472s) [noise=true , address=145.79.227.41:4087 , piece_id=SI6ATCZXAKM5WHROUXKWV3UU7TS2SY65X6UKIIF5RSACT3QHTBOA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.270161989 (269h59m13.250514629s) [noise=true , address=145.79.227.49:4337 , piece_id=QDXM7ZUX6HE2WVWVBZ4KCNIIULP2TNW5V2UWJCOWJONR52UEOHIQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.272598565 (269h59m13.252951212s) [noise=true , address=145.79.227.31:4799 , piece_id=4SVPL3VZQTGBUUWQ5555JFFO65KJQBH6P5MGWKV5AQOUQHY6JALA, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:33.280337514 (269h59m13.260690154s) [noise=true , address=145.79.227.28:4725 , piece_id=ZC2Q3OTSTKM3T7263ACUMWCDDXMCGDJ2RAI2JLYP42ZM27SVEX3Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:37.051462201 (269h59m17.031814842s) [noise=true , address=145.79.227.33:4871 , piece_id=IR43CTU2I3FHT3VVQE2LBUOSF5P6HHIS5D3WLVSRVZEYWCABI24Q, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:27:44.649153667 (269h59m24.629506307s) [noise=true , address=145.79.227.7:4293 , piece_id=DQN22FQYKAFTQCZQL3ACRNBFZOHOUUQBXDAN6KLMBKHL2PEZQ7RQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.252814639 (269h59m40.233167266s) [noise=true , address=145.79.227.51:4411 , piece_id=UN64G4U2ZCHUE7AHUBXANMYMP7TIUG7OB6NNNI2M4C7YJH5YEUTQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896047269 (269h59m40.87639991s) [noise=true , address=145.79.227.50:4373 , piece_id=YLGVY7JQARUU7ATYRMNZA426STJLT3RAP56HJYDKMKE2PGDC77TQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: <nil> Dec 16 21:28:00.896076543 (269h59m40.876429187s) [seg_pos={0 0}, upload=237383]: Segment reached optimal threshold of 13 pieces. Dec 16 21:28:00.896336348 (269h59m40.876688995s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896345920 (269h59m40.876698559s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHGSNJ4UOH7UNYIGA7BLBDMRF4K6RAUJ2J4EE7X5YVUL3LR2KPQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896682605 (269h59m40.877035248s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 16 21:28:00.896694473 (269h59m40.877047122s) [noise=true , address=145.79.227.35:4481 , piece_id=QXAGSIMLU62XIDSGSFYH6RH2MNTK4JWAQIVH42NTN25OBLA57VYQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896742281 (269h59m40.877094904s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): internal: context canceled Dec 16 21:28:00.896750506 (269h59m40.877103151s) [noise=true , address=145.79.227.47:4265 , piece_id=XHHK72XHQB2OXUIXND3P3DARRG2MCLKRX2ABWPRXFRUHDIEAIXUQ, seg_pos={0 0}, upload=237383]: Long tail cancellation due to optimal threshold Dec 16 21:28:00.896773228 (269h59m40.877125872s) [seg_pos={0 0}, upload=237383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 21:28:00.896870465 (269h59m40.877223105s) [upload=237383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 21:28:00.896877199 (269h59m40.877229842s) [upload=237383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 21:28:01.039964881 (269h59m41.020317525s) [upload=237383]: Done uploading segments... Dec 16 21:28:01.040007828 (269h59m41.020360481s) [upload=237383]: Upload finished. err: <nil> Duration: 30.005680585s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.054270076s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:27:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.054299152s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.044607883s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:26:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.044636178s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 986.588016ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:26:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 986.633443ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.000966384s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.000999928s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 992.170334ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:25:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 992.196813ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.078721809s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:24:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.078813258s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 971.565227ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:23:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 971.6221ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.031173945s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:22:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.031203136s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.002749941s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:22:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.002781258s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.030223619s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:21:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.030255891s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.043822961s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:20:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.043858036s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.082323193s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:19:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.08234692s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.076264908s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:18:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.07630022s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.024612826s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:18:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.024642481s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.082175352s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:17:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.082197143s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.029163399s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:17:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.029198308s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.104022775s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:16:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.104055361s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 964.554048ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:15:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 964.583702ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 992.053954ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 992.079818ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 804.900455ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:13:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 804.923789ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 952.888991ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:13:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 952.912756ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.039643419s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:12:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.039667399s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 995.56524ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:11:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 995.595135ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.043774166s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:10:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.043795066s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 953.822668ms trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:09:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 953.842618ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.026157641s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:09:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.026190941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.04626662s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:08:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.046307665s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.024426713s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.024449483s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.037551856s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:06:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.037588033s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.190182257s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:05:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.190214426s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.051359564s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:04:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.05182598s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.082558509s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:03:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.082569388s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.046701528s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:03:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.046732582s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.007036158s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:02:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.007066295s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.03172269s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.031749243s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.057140766s trace: https://jaeger.dev.storj.io/trace/60f701d339172dbe FAIL (2025-12-16 17:00:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.057177597s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)