Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 2.451773921s FAIL (2025-08-17 13:17:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.451854557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXX.X.........X.....] duration: 2.782702088s FAIL (2025-08-17 13:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:13:09.150950234 (211h47m31.110047467s) [upload=220493]: Starting upload Aug 17 13:13:09.151030369 (211h47m31.110127623s) [upload=220493]: Done starting upload Aug 17 13:13:09.151161044 (211h47m31.11025829s) [upload=220493]: Uploading segments... Aug 17 13:13:09.151432451 (211h47m31.110529687s) [upload=220493]: Got next segment. Inline: false Aug 17 13:13:09.151484231 (211h47m31.110581478s) [upload=220493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:13:09.151490012 (211h47m31.110587248s) [upload=220493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:13:09.239305828 (211h47m31.198403083s) [upload=220493]: Next segment err: metaclient: Too Many Requests Aug 17 13:13:09.239329027 (211h47m31.198426273s) [upload=220493]: Done uploading segments... Aug 17 13:13:09.239345887 (211h47m31.198443134s) [upload=220493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.782742335s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXX.X.........X....] duration: 2.893225273s FAIL (2025-08-17 13:16:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:13:09.150950234 (211h47m31.110047467s) [upload=220493]: Starting upload Aug 17 13:13:09.151030369 (211h47m31.110127623s) [upload=220493]: Done starting upload Aug 17 13:13:09.151161044 (211h47m31.11025829s) [upload=220493]: Uploading segments... Aug 17 13:13:09.151432451 (211h47m31.110529687s) [upload=220493]: Got next segment. Inline: false Aug 17 13:13:09.151484231 (211h47m31.110581478s) [upload=220493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:13:09.151490012 (211h47m31.110587248s) [upload=220493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:13:09.239305828 (211h47m31.198403083s) [upload=220493]: Next segment err: metaclient: Too Many Requests Aug 17 13:13:09.239329027 (211h47m31.198426273s) [upload=220493]: Done uploading segments... Aug 17 13:13:09.239345887 (211h47m31.198443134s) [upload=220493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.893255434s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X..........X..............................X.............] duration: 814.76033ms FAIL (2025-08-17 13:16:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 814.801969ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 2.549374198s FAIL (2025-08-17 13:16:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.549408681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 2.334950888s FAIL (2025-08-17 13:16:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.334989295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........X..............................X............] duration: 840.086829ms FAIL (2025-08-17 13:15:49 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 840.192997ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.........................................XXX.X.........X...] duration: 2.349190277s FAIL (2025-08-17 13:15:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:13:09.150950234 (211h47m31.110047467s) [upload=220493]: Starting upload Aug 17 13:13:09.151030369 (211h47m31.110127623s) [upload=220493]: Done starting upload Aug 17 13:13:09.151161044 (211h47m31.11025829s) [upload=220493]: Uploading segments... Aug 17 13:13:09.151432451 (211h47m31.110529687s) [upload=220493]: Got next segment. Inline: false Aug 17 13:13:09.151484231 (211h47m31.110581478s) [upload=220493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:13:09.151490012 (211h47m31.110587248s) [upload=220493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:13:09.239305828 (211h47m31.198403083s) [upload=220493]: Next segment err: metaclient: Too Many Requests Aug 17 13:13:09.239329027 (211h47m31.198426273s) [upload=220493]: Done uploading segments... Aug 17 13:13:09.239345887 (211h47m31.198443134s) [upload=220493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.349225142s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 2.575978766s FAIL (2025-08-17 13:15:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.576011827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [X.....X..........X..............................X...........] duration: 833.135009ms FAIL (2025-08-17 13:15:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 833.156658ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 2.277969263s FAIL (2025-08-17 13:15:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.278004219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X.....X..........X..............................X..........] duration: 1.016078283s FAIL (2025-08-17 13:15:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 1.016109082s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.........................................XXX.X.........X..] duration: 3.000989472s FAIL (2025-08-17 13:14:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:13:09.150950234 (211h47m31.110047467s) [upload=220493]: Starting upload Aug 17 13:13:09.151030369 (211h47m31.110127623s) [upload=220493]: Done starting upload Aug 17 13:13:09.151161044 (211h47m31.11025829s) [upload=220493]: Uploading segments... Aug 17 13:13:09.151432451 (211h47m31.110529687s) [upload=220493]: Got next segment. Inline: false Aug 17 13:13:09.151484231 (211h47m31.110581478s) [upload=220493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:13:09.151490012 (211h47m31.110587248s) [upload=220493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:13:09.239305828 (211h47m31.198403083s) [upload=220493]: Next segment err: metaclient: Too Many Requests Aug 17 13:13:09.239329027 (211h47m31.198426273s) [upload=220493]: Done uploading segments... Aug 17 13:13:09.239345887 (211h47m31.198443134s) [upload=220493]: Upload finished. err: metaclient: Too Many Requests Duration: 3.001021855s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 2.843889955s FAIL (2025-08-17 13:14:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.843912725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 2.343900013s FAIL (2025-08-17 13:14:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.343935294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X.....X..........X..............................X.........] duration: 854.613503ms FAIL (2025-08-17 13:14:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 854.653582ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 2.376410261s FAIL (2025-08-17 13:14:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.376440946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.........................................XXX.X.........X.] duration: 2.375510598s FAIL (2025-08-17 13:13:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:13:09.150950234 (211h47m31.110047467s) [upload=220493]: Starting upload Aug 17 13:13:09.151030369 (211h47m31.110127623s) [upload=220493]: Done starting upload Aug 17 13:13:09.151161044 (211h47m31.11025829s) [upload=220493]: Uploading segments... Aug 17 13:13:09.151432451 (211h47m31.110529687s) [upload=220493]: Got next segment. Inline: false Aug 17 13:13:09.151484231 (211h47m31.110581478s) [upload=220493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:13:09.151490012 (211h47m31.110587248s) [upload=220493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:13:09.239305828 (211h47m31.198403083s) [upload=220493]: Next segment err: metaclient: Too Many Requests Aug 17 13:13:09.239329027 (211h47m31.198426273s) [upload=220493]: Done uploading segments... Aug 17 13:13:09.239345887 (211h47m31.198443134s) [upload=220493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.375539669s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 2.330757656s FAIL (2025-08-17 13:13:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.3307928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.........................................XXX.X.........X] duration: 355.263213ms FAIL (2025-08-17 13:13:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:13:09.150950234 (211h47m31.110047467s) [upload=220493]: Starting upload Aug 17 13:13:09.151030369 (211h47m31.110127623s) [upload=220493]: Done starting upload Aug 17 13:13:09.151161044 (211h47m31.11025829s) [upload=220493]: Uploading segments... Aug 17 13:13:09.151432451 (211h47m31.110529687s) [upload=220493]: Got next segment. Inline: false Aug 17 13:13:09.151484231 (211h47m31.110581478s) [upload=220493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:13:09.151490012 (211h47m31.110587248s) [upload=220493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:13:09.239305828 (211h47m31.198403083s) [upload=220493]: Next segment err: metaclient: Too Many Requests Aug 17 13:13:09.239329027 (211h47m31.198426273s) [upload=220493]: Done uploading segments... Aug 17 13:13:09.239345887 (211h47m31.198443134s) [upload=220493]: Upload finished. err: metaclient: Too Many Requests Duration: 355.304663ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 2.57907369s FAIL (2025-08-17 13:13:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.579110222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 2.3483542s FAIL (2025-08-17 13:12:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.348388753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...X.....X..........X..............................X........] duration: 1.210332315s FAIL (2025-08-17 13:12:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 1.210372959s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...X.........................................XXX.X.........] duration: 2.535268409s FAIL (2025-08-17 13:12:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.535303386s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 2.242089185s FAIL (2025-08-17 13:12:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.242124239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...X.........................................XXX.X........] duration: 2.641161393s FAIL (2025-08-17 13:12:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.641199415s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....X.....X..........X..............................X.......] duration: 998.720743ms FAIL (2025-08-17 13:11:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 998.753521ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 2.608513281s FAIL (2025-08-17 13:11:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.608549177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 2.36576775s FAIL (2025-08-17 13:11:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.365799645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 2.49862256s FAIL (2025-08-17 13:11:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.498656211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X.....X..........X..............................X......] duration: 1.693314256s FAIL (2025-08-17 13:11:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 1.693344308s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X.....X..........X..............................X.....] duration: 917.951301ms FAIL (2025-08-17 13:10:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 917.984814ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...X.........................................XXX.X.......] duration: 2.554270638s FAIL (2025-08-17 13:10:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.554302783s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 2.296896395s FAIL (2025-08-17 13:10:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.296929378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...X.........................................XXX.X......] duration: 2.550790748s FAIL (2025-08-17 13:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.550814612s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......X.....X..........X..............................X....] duration: 823.329057ms FAIL (2025-08-17 13:10:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 823.364635ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXXX.....] duration: 2.522320542s FAIL (2025-08-17 13:10:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.522353108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXXX.....] duration: 2.478568018s FAIL (2025-08-17 13:09:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.478599188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...X.........................................XXX.X.....] duration: 2.806914613s FAIL (2025-08-17 13:09:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.806940579s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 2.671446672s FAIL (2025-08-17 13:09:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.671502147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........X.....X..........X..............................X...] duration: 1.105974221s FAIL (2025-08-17 13:09:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 1.106001299s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 2.557459677s FAIL (2025-08-17 13:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.557497003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...X.........................................XXX.X....] duration: 2.939229714s FAIL (2025-08-17 13:08:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.93925064s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...X.........................................XXX.X...] duration: 4.169063841s FAIL (2025-08-17 13:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 4.16914929s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 2.575472158s FAIL (2025-08-17 13:08:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.575497138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 3.530178864s FAIL (2025-08-17 13:08:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.530232298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....X..........X..............................X..] duration: 872.218852ms FAIL (2025-08-17 13:08:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 17 13:05:53.292389310 (211h40m15.251486567s) [upload=220370]: Starting upload Aug 17 13:05:53.292477664 (211h40m15.251574912s) [upload=220370]: Done starting upload Aug 17 13:05:53.292569033 (211h40m15.251666269s) [upload=220370]: Uploading segments... Aug 17 13:05:53.292624407 (211h40m15.251721661s) [upload=220370]: Got next segment. Inline: true Aug 17 13:05:53.292630660 (211h40m15.251727906s) [upload=220370]: Waiting for error group managing segments... Aug 17 13:05:53.292698624 (211h40m15.25179587s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:53.292705778 (211h40m15.251803025s) [upload=220370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:05:53.292710869 (211h40m15.251808104s) [upload=220370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:05:53.454358020 (211h40m15.413455265s) [upload=220370]: Done uploading segments... Aug 17 13:05:53.454376585 (211h40m15.413473816s) [upload=220370]: Upload finished. err: <nil> Duration:* 872.254146ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X......X...X.........................................XXX.X..] duration: 3.02462618s FAIL (2025-08-17 13:07:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 17 13:05:15.231132314 (211h39m37.190229563s) [upload=220364]: Starting upload Aug 17 13:05:15.231201354 (211h39m37.190298599s) [upload=220364]: Done starting upload Aug 17 13:05:15.231310585 (211h39m37.190407831s) [upload=220364]: Uploading segments... Aug 17 13:05:15.231456676 (211h39m37.190553911s) [upload=220364]: Got next segment. Inline: false Aug 17 13:05:15.231498127 (211h39m37.190595363s) [upload=220364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:15.231503865 (211h39m37.1906011s) [upload=220364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:15.471516165 (211h39m37.430613434s) [seg_pos={0 0}, upload=220364]: Begin upload segment... Aug 17 13:05:15.471587514 (211h39m37.430684759s) [seg_pos={0 0}, upload=220364]: Done begin upload segment. Aug 17 13:05:15.471609049 (211h39m37.430706301s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.471622240 (211h39m37.43071949s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472020819 (211h39m37.43111807s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472124337 (211h39m37.431221588s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472272746 (211h39m37.431369991s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472443582 (211h39m37.431540827s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472557340 (211h39m37.431654595s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472720940 (211h39m37.431818191s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.472906440 (211h39m37.432003672s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473034316 (211h39m37.432131565s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.473331425 (211h39m37.432428674s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474347101 (211h39m37.43344435s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474663162 (211h39m37.43376041s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.474927269 (211h39m37.434024514s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475196383 (211h39m37.434293627s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475458683 (211h39m37.434555935s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.475825805 (211h39m37.43492305s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:15.478085146 (211h39m37.437182398s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Uploading piece... Aug 17 13:05:16.059132184 (211h39m38.018229426s) [upload=220364]: Next returned nil segment Aug 17 13:05:16.059155704 (211h39m38.018252958s) [upload=220364]: Waiting for error group managing segments... Aug 17 13:05:16.199283365 (211h39m38.158380597s) [noise=true , address=77.90.31.11:4565 , piece_id=4UVN557I2BBABXEL4DPYDYPOW5VVEXTQ3HYHWJ3X75NORAF7UGSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.211255621 (211h39m38.170352873s) [noise=true , address=79.127.219.48:4103 , piece_id=MBCIDSMYQ3SMSJ7EXCLOLY2D5B2CDUAH3EOWIJ6FYK3Z3G7ZBOEA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.238501913 (211h39m38.197599167s) [noise=true , address=77.90.31.21:4908 , piece_id=QP5VO6LNLP3SPYL6MZN3W4PUTMBZV6JA7KA7PQKNPTJW6GAOVYSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.239391183 (211h39m38.198488432s) [noise=true , address=45.91.116.7:4041 , piece_id=I25VXDBCAJGZXINGWNRNR6EU3S74QE36HT4WDWIZABC7NUCFVVGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.248341533 (211h39m38.207438788s) [noise=true , address=77.90.31.15:4709 , piece_id=WP3Y4IOFI63AVFMSKCZ5P5IQ2YUSEPHTKJYFHCGB4ERXXTB4D56A, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.264727312 (211h39m38.22382457s) [noise=true , address=77.90.31.12:4596 , piece_id=5QBGKCRL6ATG2FRF6QRWWCLKH7UFOFADRQWBDB7XNG6LPQ3CGWGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.397663238 (211h39m38.356760479s) [noise=true , address=79.127.220.101:4069 , piece_id=OE7EWTLTAOTATKU7BZEZCVFDOGSHNKLHFSCBHSU4G7BAA5CRJDGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.518964003 (211h39m38.478061242s) [noise=true , address=79.127.201.215:4339 , piece_id=ZT7Z5MNYD5AIIJYTXXQ33AC2P6AELCGSIRIKVXC6PTYCOXAEEZFA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:05:16.742044903 (211h39m38.701142141s) [noise=true , address=66.23.193.175:4564 , piece_id=FF7VEWFTCXRLY4KIB6EIZE5P7K5RL2JQA2N752BKI5I66CM32TRA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: <nil> Aug 17 13:06:14.960909529 (211h40m36.920006767s) [noise=true , address=104.36.82.23:4472 , piece_id=RZUTENAZFC7Y5SOIFVOHANBT77ZZRC6OEX3LRCBWE2R6Q2CNG6AQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct, address:104.36.82.23:4472): protocol: write tcp 139.59.144.22:53166->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.960996733 (211h40m36.920093969s) [noise=true , address=104.36.82.11:4145 , piece_id=VPBQJJCNS3HF2DXRUTSRYV34UIFNVRY7NRSJELOLQKCKDTT62PSQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: write tcp 139.59.144.22:35182->104.36.82.11:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961060154 (211h40m36.920157394s) [noise=true , address=104.36.82.59:4328 , piece_id=WQUSRI43K5DWZB3BEGDCTR67P6GSRQCEYUV6DPUMSSTHF5JTG4IQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY, address:104.36.82.59:4328): protocol: write tcp 139.59.144.22:58426->104.36.82.59:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961090044 (211h40m36.920187283s) [noise=true , address=104.36.82.60:4345 , piece_id=D2N5FIYQRF5OZL6HUFEWXGN3Y7V6BDHHRZX5XJXXHAH6FFEBM2GA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:104.36.82.60:4345): protocol: write tcp 139.59.144.22:60854->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961135671 (211h40m36.920232924s) [noise=true , address=104.36.82.5:4855 , piece_id=CMIMOGLPJMA4I6IT5Y3UIF5UMDOSQPEWY7CRUM2U6YNZD7LFLQ4Q, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 139.59.144.22:59808->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961162492 (211h40m36.920259729s) [noise=true , address=104.36.82.17:4331 , piece_id=4BITIPU4KQTZC4C3F4DPZ2B36ONVRSAZCOET2M55ETZEUCYP2RGQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: write tcp 139.59.144.22:49526->104.36.82.17:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961192413 (211h40m36.920289649s) [noise=true , address=104.36.82.58:4268 , piece_id=COQ3BBLNLSJWWDGHE5ZS2KW22V2JMLJSSKKM4IGQLACCTLYCPPSA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:104.36.82.58:4268): protocol: write tcp 139.59.144.22:35378->104.36.82.58:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961221557 (211h40m36.920318801s) [noise=true , address=104.36.82.8:4003 , piece_id=J5XKSLSVBLZQBIM34KR4GL3AQVDHZFPBIT55NAJUNOTLXAGBUPZQ, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 139.59.144.22:34818->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:14.961310318 (211h40m36.920407554s) [noise=true , address=104.36.82.3:4770 , piece_id=INPELIXUHM4GSEMG3X3MXP4SDQHJKCTUDCFKKHWTLDQJM56S3IWA, seg_pos={0 0}, upload=220364]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:06:14.961331009 (211h40m36.920428255s) [seg_pos={0 0}, upload=220364]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 17 13:06:15.094067555 (211h40m37.053164805s) [upload=220364]: Done uploading segments... Aug 17 13:06:15.094095859 (211h40m37.053193091s) [upload=220364]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.024663694s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 2.367503986s FAIL (2025-08-17 13:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 13:04:48.671168571 (211h39m10.630265822s) [upload=220354]: Starting upload Aug 17 13:04:48.671289154 (211h39m10.630386405s) [upload=220354]: Done starting upload Aug 17 13:04:48.671375030 (211h39m10.630472267s) [upload=220354]: Uploading segments... Aug 17 13:04:48.671685291 (211h39m10.630782528s) [upload=220354]: Got next segment. Inline: false Aug 17 13:04:48.671859547 (211h39m10.630956792s) [upload=220354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:04:48.671868198 (211h39m10.630965451s) [upload=220354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:04:48.784496066 (211h39m10.743593311s) [seg_pos={0 0}, upload=220354]: Begin upload segment... Aug 17 13:04:48.784546278 (211h39m10.743643523s) [seg_pos={0 0}, upload=220354]: Done begin upload segment. Aug 17 13:04:48.784576969 (211h39m10.743674214s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784591947 (211h39m10.743689244s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.784926858 (211h39m10.744024104s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785024612 (211h39m10.744121858s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785177382 (211h39m10.744274627s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785386459 (211h39m10.744483708s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785426090 (211h39m10.744523335s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.785675714 (211h39m10.744772959s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786030292 (211h39m10.745127541s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786370888 (211h39m10.745468139s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786497361 (211h39m10.745594613s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.786709369 (211h39m10.745806618s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787016861 (211h39m10.746114097s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787272873 (211h39m10.746370129s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.787671377 (211h39m10.746768629s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:48.788114039 (211h39m10.747211295s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Uploading piece... Aug 17 13:04:49.396438949 (211h39m11.355536206s) [upload=220354]: Next returned nil segment Aug 17 13:04:49.396459727 (211h39m11.355556984s) [upload=220354]: Waiting for error group managing segments... Aug 17 13:04:49.548390642 (211h39m11.507487888s) [noise=true , address=77.90.31.11:4553 , piece_id=AG7MIKIR6GZIAHY7MHORQRCNQCYR7IFHMFPTXAU54UJME5JPAXZA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.555382402 (211h39m11.514479641s) [noise=true , address=45.91.116.11:4118 , piece_id=FMR757UBDMQR6KY5NXBJDCIEJWVMVDXZVKKQ6WJSCB6QJPXSMFMQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.568461124 (211h39m11.527558355s) [noise=true , address=45.91.116.8:4047 , piece_id=N2MAQURNFGRJODRX7NNIWR7FBWMBY6OEIMS67YDRU37NJBIGQEFQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.594216288 (211h39m11.553313519s) [noise=true , address=45.91.116.4:4940 , piece_id=2WVMQJUCVS4RISINDJVSNLUPEQTM5A5UUVEKH4LBPUNV37OZ4GHQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.596651299 (211h39m11.555748557s) [noise=true , address=45.91.116.14:4189 , piece_id=PZOLHTZEUD5ZBOPZZCMYSPBHBF3HYIQOJIQQDWKW3BXQBH35KOPQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.600004059 (211h39m11.559101306s) [noise=true , address=45.91.116.5:4297 , piece_id=KGKAAJRVI4UUO7GL36NKH55IPLZM7CYBNW2D632OTIW2XRCXFL4A, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.619246354 (211h39m11.578343598s) [noise=true , address=45.91.116.6:4974 , piece_id=D2UBLOQZFV5Y65PPNOYZANFQYJA3VW5MYHROACSEDU3N4KBM2ECQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.622590623 (211h39m11.581687861s) [noise=true , address=45.91.116.5:4299 , piece_id=C7VTE7JWLJJNPIE7FFFUXR46VFSIQKI7OJJWWAIXBDHCKPTFMGOQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.633199644 (211h39m11.592296882s) [noise=true , address=77.90.31.13:4628 , piece_id=VUCK3GUU7RKFAYILIYPNGHCZAXYK743ZBN35LVBZMGZGLEM73X5Q, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:04:49.638567248 (211h39m11.597664486s) [noise=true , address=77.90.31.21:4879 , piece_id=NIQXMIEXQFZZ42M5MZACGECP5ZBR32MSZGHNFHJ6QENFN2DWEZSQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: <nil> Aug 17 13:05:18.402418747 (211h39m40.361515993s) [noise=true , address=104.36.82.13:4186 , piece_id=RPF3NP62OFMDLTDNL5QIBV2IB2SM5AGXSG32QWDGMFGRSNLFACCA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402498626 (211h39m40.361595873s) [noise=true , address=104.36.82.4:4801 , piece_id=U4NVLZC6LLZFIZE3GNN72R5E7JVPPDR66FEJZ5QCRBIICLPBKNHA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:18.402587513 (211h39m40.361684745s) [noise=true , address=104.36.82.8:4969 , piece_id=QBKZULXXBNFTBZE3GRFX6WKLFS4B3UEZDN6D4O3T74AO5R5JR5EA, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402653738 (211h39m40.361750984s) [noise=true , address=104.36.82.7:4909 , piece_id=MZKLM66QYEALDLPVAECEMRFNTCLV3SXOSNHMJB5CKMPMBLHE4AEQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402768644 (211h39m40.361865891s) [noise=true , address=104.36.82.11:4114 , piece_id=72DKJ6BLHEEOEMMUI3NLYYYRUUABEXDOJK2I3Y3M3UGEWMGDR3SQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402931362 (211h39m40.362028611s) [noise=true , address=104.36.82.2:4730 , piece_id=6IMVJQPWAS7URN7QYWPYCD2RZK2C5GZCQGUSJ6RUOL3XJO3O7HZQ, seg_pos={0 0}, upload=220354]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Aug 17 13:05:18.402953112 (211h39m40.362050358s) [seg_pos={0 0}, upload=220354]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 13:05:18.518203938 (211h39m40.477301184s) [upload=220354]: Done uploading segments... Aug 17 13:05:18.518239058 (211h39m40.477336313s) [upload=220354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.367556551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 2.796386734s FAIL (2025-08-17 13:07:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== s={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860424250 (211h39m24.819521505s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860443782 (211h39m24.819541042s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860834658 (211h39m24.819932015s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.860891189 (211h39m24.819988472s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861295596 (211h39m24.820392854s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861308085 (211h39m24.820405343s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861721034 (211h39m24.820818312s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.861738713 (211h39m24.82083598s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862205183 (211h39m24.821302442s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862325407 (211h39m24.821422696s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862620160 (211h39m24.82171753s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:02.862835186 (211h39m24.821932443s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Uploading piece... Aug 17 13:05:03.436772519 (211h39m25.395869766s) [upload=220356]: Next returned nil segment Aug 17 13:05:03.436799550 (211h39m25.395896811s) [upload=220356]: Waiting for error group managing segments... Aug 17 13:05:03.580591291 (211h39m25.539688524s) [noise=true , address=45.91.116.12:4228 , piece_id=MTH3FCTHOBUONI3ATWGLX7T422COVZAR44ES2SASV6IPS4TT32NA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.649197127 (211h39m25.608294368s) [noise=true , address=77.90.31.15:4703 , piece_id=P7LMDBYDZ72IDHBPVYMAUKQAF3PEHYZ7TWL72VGCNLBDF56GAU4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.678085375 (211h39m25.637182615s) [noise=true , address=77.90.31.14:4676 , piece_id=XTWEFWPNQHGH3HBL42U2JVOYHI34SZQ3BAJBUW5OHAYQG73J7N6Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.690242999 (211h39m25.649340239s) [noise=true , address=77.90.31.11:4563 , piece_id=SMDGS5CXFE6LF7CAEENVWLSWTSXLBOBX7QKPZVXFWRHD4I6VC7GA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:03.948496632 (211h39m25.907593866s) [noise=true , address=79.127.201.215:4354 , piece_id=RHT2PSXNWPW6DX5FKBGITLOFMJ36KMCFRHV6AP5MHIGQCYIXR6NQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:05:04.108115883 (211h39m26.067213141s) [noise=true , address=66.23.193.175:4555 , piece_id=6EKCGUSQSLFHFXNUL6BDMMG7P2W4M3ETJ4D5XNJC5V4QHYLS37UQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: <nil> Aug 17 13:06:02.389131075 (211h40m24.348228332s) [noise=true , address=104.36.82.4:4821 , piece_id=SZQ4ODJ6IBPXZPPLMZZA7533MZBN64C5NQ5S2NHNYPFQVDNAJ6LQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389262413 (211h40m24.348359673s) [noise=true , address=104.36.82.39:4756 , piece_id=XCWABRWTJ4MGGZKVMEYEFX5TEQELEPN6YBA33HVVU75VF2Z6ZZPQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389582130 (211h40m24.348679388s) [noise=true , address=104.36.82.51:4079 , piece_id=4JGPKABKRRQU4DAIMRLQZAFRHA7OS45MFYQBTA6TDQTSXKRSKFPA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: write tcp 139.59.144.22:46906->104.36.82.51:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389643456 (211h40m24.348740693s) [noise=true , address=104.36.82.21:4506 , piece_id=IO7ZU2CMWODCGXUKHA2NORHMNAZCD24RWEZEKYEIZIN24CK6XLRQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389692881 (211h40m24.348790138s) [noise=true , address=104.36.82.23:4462 , piece_id=3ML37MP2TQYQDRUDACGK3LVMZPY6CQ6CDETI3UCHRUQ2EOOPHXJA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389734487 (211h40m24.348831728s) [noise=true , address=104.36.82.2:4762 , piece_id=BTP6XPRTEVYKQEXKZYROAV7WF3GPMGPDWCJNCYBR4SBTINE3JYXA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: write tcp 139.59.144.22:36810->104.36.82.2:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389842740 (211h40m24.348939975s) [noise=true , address=104.36.82.60:4352 , piece_id=LYTVTLNMO7DQX6F3ZSEPWROQTJPQRST4IAXSNH24SCHU4QNX4WCA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: write tcp 139.59.144.22:42132->104.36.82.60:4352: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389939930 (211h40m24.349037163s) [noise=true , address=104.36.82.40:4949 , piece_id=JO4YEFXT72PQESTS5XAJRB5744GKOQIUB2L2QVA7HM6KML3MFN7A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.389986244 (211h40m24.349083492s) [noise=true , address=104.36.82.58:4275 , piece_id=FONUFYYHTF5A5OAMTO2JH4XLWWKTXR4MFN6MYKFCIJ4EPFII3P4Q, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12Mb3s64JoVf1TbMuMRpT67xzAN4YSE9txxRnjomuvZUpSj9tt3, address:104.36.82.58:4275): protocol: write tcp 139.59.144.22:34692->104.36.82.58:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390006984 (211h40m24.349104216s) [noise=true , address=104.36.82.36:4626 , piece_id=72QP3R24SKEOGZAY2FJPMVYNQVMIKAEFNZ7TZRD2X4RAR5WHJE5A, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: rpc: dial tcp 104.36.82.36:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390056341 (211h40m24.34915358s) [noise=true , address=104.36.82.5:4844 , piece_id=S6FQV3NHEMEUQFH7ARLE76XJJBQ4WE22ZGI3CPLWGFE3QBYCNOTA, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa, address:104.36.82.5:4844): protocol: write tcp 139.59.144.22:34632->104.36.82.5:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390105755 (211h40m24.349203011s) [noise=true , address=104.36.82.9:4025 , piece_id=I6MSLKGJY5N2CKRVKAKXVG6UAZN65CBQHYV6DUU5IGY5SVZZVNDQ, seg_pos={0 0}, upload=220356]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:37148->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:02.390162167 (211h40m24.349259427s) [seg_pos={0 0}, upload=220356]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 17 13:06:02.490382093 (211h40m24.449479331s) [upload=220356]: Done uploading segments... Aug 17 13:06:02.490422033 (211h40m24.449519267s) [upload=220356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.796531616s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)