Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 2.188475509s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:27:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.188500279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 2.204350489s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:26:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.20446339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 1.612276324s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:25:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.612386173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 1.592880574s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:24:44 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.592912186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 1.59866678s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:24:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.598695676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 1.572381259s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:22:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.572407786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 2.210968317s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:22:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.210996788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 1.623216278s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:22:00 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.623257972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 1.943646776s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:21:34 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.94367235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.705493976s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:20:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.705519671s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.562360915s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:20:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.562396991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 1.736950736s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:19:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.736978228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 1.563843613s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:18:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.563876009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.66483566s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:17:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.664928271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 2.207568436s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:16:34 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.20759213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001493728s trace: https://jaeger.dev.storj.io/trace/7e6f60b4ca1938a4 FAIL (2026-03-07 17:16:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001552603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.150565998s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:37:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.150594746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.042714927s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:36:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.042746841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.166420074s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:35:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.166444659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.0990305s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:34:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.099056377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.15011733s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:33:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.150209012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.102360772s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.102385634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.209064177s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:31:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.209085888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.226062955s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:30:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.2260906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.076857952s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.076883708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.045072387s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:28:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.045099291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.013156622s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:28:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.013185361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.941952761s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.942040445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.148573204s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:27:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.14869025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.062524724s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:26:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.062548277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.171548531s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:25:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.17157784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.675321885s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:24:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.675345281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.056651685s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.056679059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.999715951s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:23:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.999739626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 3.190977309s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:22:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.191014381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.032686407s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:21:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.032712392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.172761839s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:21:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.172808071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 3.030801412s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:21:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.030828481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.021879577s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:20:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.021913359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.242731998s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:19:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.242758894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.129070256s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.129092093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.935874377s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.935901901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.950370482s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:17:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.950416401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.981953464s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:16:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.982085362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.22089936s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:15:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.220924823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.212643429s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:14:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.212685664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.054798181s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:13:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.054825188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.167278771s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:12:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.167309103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.083164166s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:12:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.08319375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.049655863s trace: https://jaeger.dev.storj.io/trace/5821c77b392ac5d0 FAIL (2026-03-07 15:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 7 14:50:01.350487446 (27h49m21.162979393s) [upload=26727]: Starting upload Mar 7 14:50:01.350593882 (27h49m21.163085839s) [upload=26727]: Done starting upload Mar 7 14:50:01.350950467 (27h49m21.163442416s) [upload=26727]: Uploading segments... Mar 7 14:50:01.352011362 (27h49m21.164503317s) [upload=26727]: Got next segment. Inline: false Mar 7 14:50:01.352116444 (27h49m21.164608403s) [upload=26727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 7 14:50:01.352126376 (27h49m21.164618346s) [upload=26727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 7 14:50:01.471811963 (27h49m21.284303921s) [seg_pos={0 0}, upload=26727]: Begin upload segment... Mar 7 14:50:01.471906845 (27h49m21.284398786s) [seg_pos={0 0}, upload=26727]: Done begin upload segment. Mar 7 14:50:01.471980882 (27h49m21.284472843s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.479456010 (27h49m21.291947971s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.480653424 (27h49m21.293145381s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.481377529 (27h49m21.293869485s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482041197 (27h49m21.294533156s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.482692493 (27h49m21.295184451s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.483373694 (27h49m21.295865653s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484302669 (27h49m21.296794628s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.484906375 (27h49m21.29739833s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.485465610 (27h49m21.297957569s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486008756 (27h49m21.298500714s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.486678561 (27h49m21.299170518s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.487457872 (27h49m21.299949831s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.488356527 (27h49m21.300848487s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.489170646 (27h49m21.301662606s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.490272433 (27h49m21.302764393s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491015469 (27h49m21.303507426s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:01.491547118 (27h49m21.304039069s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Uploading piece... Mar 7 14:50:02.133677856 (27h49m21.946169812s) [upload=26727]: Next returned nil segment Mar 7 14:50:02.133701063 (27h49m21.946193022s) [upload=26727]: Waiting for error group managing segments... Mar 7 14:50:02.309593208 (27h49m22.122085169s) [noise=true , address=45.91.116.6:4002 , piece_id=GKFYRDBWNNM2VW3QX66UZC4YLONTU6DUHAAPJ4IDJA25MJY265WA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.318721054 (27h49m22.131213013s) [noise=true , address=45.91.116.14:4218 , piece_id=OAQR2EOSRXZIFRHRAS7YQJLK7PZ5Y22NZ7PL43C4JAVHI7CKO5NQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.323573340 (27h49m22.136065293s) [noise=true , address=145.79.227.6:4289 , piece_id=6PKQUENUBBWU53K5PCDFNNZP3FES3NY42GZ6EKIDPUUZ4GYXT6ZQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.698731316 (27h49m22.511223254s) [noise=true , address=167.17.72.68:4657 , piece_id=6HHBZKX672MST3UHQGOMXXRQWGVEEQLOB7ITNE34N2CLU3NXNLQQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.781861271 (27h49m22.594353224s) [noise=true , address=193.149.164.71:4311 , piece_id=PX2ZEYBUBXIQBFBM3WJBX4QZGUIXDGFGPD6R6RTU4TSI33VVXJJA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.791672309 (27h49m22.604164266s) [noise=true , address=193.149.164.70:4115 , piece_id=HZGRA2AXICAOXKVUYGZLR3ZQ7QPMU2SLWWZHLZGXYSQ2HP2SGKJQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.794109368 (27h49m22.606601321s) [noise=true , address=193.149.164.67:4007 , piece_id=EAXQ2LZOYBDCTZRWHAOZUU5CDO3RNZKCZCYNBFIKHXL2AK4QBHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.798860632 (27h49m22.611352581s) [noise=true , address=64.49.13.169:4714 , piece_id=S5SRRVHRZGJKO2USYQT6PUIINCXTS3K2JJQWMU3JWGEBSN2VNTUQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.800172020 (27h49m22.612663955s) [noise=true , address=193.149.164.66:4991 , piece_id=US5VUCQHTDQU2UYUZW7ZVIL2K2YSW5ITSXEMWQ4G6XBQCLW4B7XA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.818423259 (27h49m22.630915207s) [noise=true , address=193.149.164.69:4092 , piece_id=PMFAFQK3NLDOLBJ3RUNQFOLECC3F773O4NTG7ROEL52SXARABFEA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.834237789 (27h49m22.646729749s) [noise=true , address=167.17.72.71:4784 , piece_id=KROS3S6ZKREXOCXZIV4JK32F6B7XBOF5XYLYRBXNLO3P6KIYBZYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.848432929 (27h49m22.660924881s) [noise=true , address=64.49.13.140:4507 , piece_id=6D3J6OBNXAFHUSSQ4I7U5XRDGNH5SAJAZBMMJSNLPZPCN2A4XBCA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:50:02.849660861 (27h49m22.662152817s) [noise=true , address=64.49.13.170:4726 , piece_id=ZQMNI3UPIZN7CBP3S3LTYOIU5LGECXNYQAMX37IA43RINVKCAQWA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: <nil> Mar 7 14:51:01.048926827 (27h50m20.861418779s) [noise=true , address=77.90.31.10:4538 , piece_id=OATEHFTNCX2G7PCE2ZN3CXIGAGH7EL2IKS72E54YXSK7AHOQACOQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: rpc: dial tcp 77.90.31.10:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 7 14:51:01.049108182 (27h50m20.861600138s) [noise=true , address=77.90.31.16:4747 , piece_id=SIWCIW6KGMBJIIJMZTIDBIP57ZAL7FNN7FI5CGCJHGJPIYR6XHYA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049168468 (27h50m20.861660416s) [noise=true , address=77.90.31.19:4873 , piece_id=GJM7WT7WJTELRNWASBP44EWLJFXXC4NUB3SU3A4VS4HJMA323ILQ, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049363856 (27h50m20.861855802s) [noise=true , address=77.90.31.17:4779 , piece_id=2RIDQ437DRKDQ6ZQ4WYGWLMLDJDVCJZKMGZJUDJSKCGL3226TDLA, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049442107 (27h50m20.861934068s) [noise=true , address=77.90.31.11:4566 , piece_id=YGTTTUJJTOAP3FHFGN5MZQWDCSHAHK2MHQZBHFHWDP4ZMG6GYQ4A, seg_pos={0 0}, upload=26727]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: context deadline exceeded; context deadline exceeded Mar 7 14:51:01.049485701 (27h50m20.861977637s) [seg_pos={0 0}, upload=26727]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 7 14:51:01.161084315 (27h50m20.97357626s) [upload=26727]: Done uploading segments... Mar 7 14:51:01.161170639 (27h50m20.973662595s) [upload=26727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.049685736s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)