Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 2.066535225s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:27:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.066562457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 1.606709722s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:26:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.606737062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 1.710589425s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:25:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.710626157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.717695658s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:25:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.717802065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.638911567s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:24:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.639014887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 1.585075468s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:23:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.585155433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 1.599431118s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:23:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.599473795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.584028237s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:22:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.584058104s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.674475746s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:21:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.674505884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000865566s trace: https://jaeger.dev.storj.io/trace/64d40cf45c1bd85 FAIL (2026-03-29 00:20:00 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.000914658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.499021142s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:26:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.499050345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.444942531s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:25:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.444973542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.497811815s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.497847984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.424364325s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:24:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.424403515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.459391464s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:23:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.45942836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.444681606s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:23:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.444715078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.552873673s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.552912619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.330090408s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:21:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.330153776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.39891046s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:20:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.398969213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.326272221s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326306299s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.385226449s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:19:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.385254128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.384891799s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:18:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.384923267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.379275487s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:18:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.379304438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.413361949s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:17:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.413460069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.460300099s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.460335402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.506012722s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:15:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.506046719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.435546301s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:15:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.435577135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.641066572s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:14:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.641115669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.439550361s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.439577663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.591053834s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:13:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.591124034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.480586251s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:12:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.480638715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.441827066s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:11:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.441857878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.544294659s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.544389545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.932594798s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.932639469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.636433415s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:09:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.636468365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.409042452s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409075998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.409546387s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:08:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409581049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.436604587s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:07:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.436634858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.530387465s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:07:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.530421036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.397877541s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:06:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.398015796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.322593954s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:05:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.322624301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.339140701s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:04:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.339180548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.664848195s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:03:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.664879521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.474536075s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.474619909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.654489563s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:01:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.654752559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.488797934s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:01:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.488840885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.576965147s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:00:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.57699267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.53808449s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 15:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.538217596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.421104014s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 14:59:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421136727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.600547606s trace: https://jaeger.dev.storj.io/trace/409ed236b50eb53 FAIL (2026-03-28 14:59:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 28 14:45:35.315648730 (430h34m38.331318331s) [upload=413459]: Starting upload Mar 28 14:45:35.315733599 (430h34m38.331403199s) [upload=413459]: Done starting upload Mar 28 14:45:35.315794102 (430h34m38.331463702s) [upload=413459]: Uploading segments... Mar 28 14:45:35.316032212 (430h34m38.331701813s) [upload=413459]: Got next segment. Inline: false Mar 28 14:45:35.316200570 (430h34m38.331870171s) [upload=413459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 28 14:45:35.316206390 (430h34m38.33187599s) [upload=413459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 28 14:45:35.430724217 (430h34m38.446393826s) [seg_pos={0 0}, upload=413459]: Begin upload segment... Mar 28 14:45:35.430795561 (430h34m38.446465164s) [seg_pos={0 0}, upload=413459]: Done begin upload segment. Mar 28 14:45:35.430836859 (430h34m38.446506464s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.430848244 (430h34m38.446517846s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431376292 (430h34m38.447045883s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.431872685 (430h34m38.447542294s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432614239 (430h34m38.448283849s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.432736747 (430h34m38.448406359s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433092040 (430h34m38.448761652s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.433377489 (430h34m38.449047102s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434098354 (430h34m38.449767944s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.434769512 (430h34m38.450439123s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435075797 (430h34m38.450745407s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.435582204 (430h34m38.451251817s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436022166 (430h34m38.451691756s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436454955 (430h34m38.452124562s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.436879556 (430h34m38.452549169s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:35.437343138 (430h34m38.453012746s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Uploading piece... Mar 28 14:45:36.049495517 (430h34m39.065165118s) [upload=413459]: Next returned nil segment Mar 28 14:45:36.049527631 (430h34m39.065197233s) [upload=413459]: Waiting for error group managing segments... Mar 28 14:45:36.182328788 (430h34m39.197998388s) [noise=true , address=145.79.227.17:4973 , piece_id=ZZMYZGBYYVWXSOHDJGVKLNG6R3NMS2AZP4XMNCDNGJDE2DYADWIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182575788 (430h34m39.198245412s) [noise=true , address=145.79.227.25:4617 , piece_id=WSDGMWM7CASNFGW2RPAH7MDX6O7VXEXZYLSN45ZYB2S46LPHDGOA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.182982538 (430h34m39.198652144s) [noise=true , address=145.79.227.9:4366 , piece_id=KHTUQ2PXIXRD5Y5XIB2L3X6TMQWGZCBIS3NPOY2RS3275SCAI5BA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.183438894 (430h34m39.199108504s) [noise=true , address=45.91.116.4:4953 , piece_id=E62ZS235DKLPVBQY6FEI3DDF3DU66LDI23ISVXANBB4M5WHQEO5Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184168119 (430h34m39.199837707s) [noise=true , address=145.79.227.12:4768 , piece_id=Q4ISGVZD47GU3RPO55DR4JOKIJ4SQ6UWMQPKHJAKKPOFAUJ2PZ4Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.184865710 (430h34m39.200535314s) [noise=true , address=145.79.227.40:4051 , piece_id=6SFERDADQONW3YTOVRYHDZ7VXAOLTNZ5I3QQAS2XHG62KCNRP7TQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.185089145 (430h34m39.20075875s) [noise=true , address=145.79.227.16:4911 , piece_id=JYKGUHIZQ4U74SM5BCQGXFA37I42PYXKOWB5CQQRFDX3I2EV527Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.188553089 (430h34m39.204222694s) [noise=true , address=145.79.227.14:4843 , piece_id=QXIASECCPGVCCC36Y6NW3NTLJMKVPNKGXWNHTCCHME2HOOW72UIQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190225530 (430h34m39.205895143s) [noise=true , address=145.79.227.2:4115 , piece_id=TA4GUBLNRHKJ7DSCENFIQ5L5M7NLSJFDT4S4PVN3RRBEIIPEUYNQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.190551513 (430h34m39.206221103s) [noise=true , address=145.79.227.21:4478 , piece_id=Z75OK7DR2TUDRPTM2RMX5FCKF456FYTMEVJVDBUZESZYFE6QQUTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.191113981 (430h34m39.206783572s) [noise=true , address=145.79.227.41:4086 , piece_id=6HKTS55E7ZO6NCX45RPJD5RF5K6DRLVRQTIZMV7ATXFVR4SACDRA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:45:36.289167312 (430h34m39.304836941s) [noise=true , address=145.79.227.49:4341 , piece_id=DOBHYVALR2MNRJX2ABYYBHUASIGTPTBJ5POZAZYIGSJX6K5EJLTQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: <nil> Mar 28 14:46:05.029585012 (430h35m8.045254629s) [noise=true , address=77.90.31.15:4702 , piece_id=56EUQLUUJUASC4QNLK4WJIE53EG5MF2DPD3Z7IBF3Q5J3ERJE3BQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50726->77.90.31.15:4702: use of closed network connection Mar 28 14:46:05.030074673 (430h35m8.045744285s) [noise=true , address=79.127.219.41:4925 , piece_id=UUW3LL2MKMROAJ62NW4A66AZR337DUXIUQACIGUK2XFGV6TOC67Q, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:1kBG5b1fGn5dLzZoD8v4KG6CHStmw1QhGpqFdtGsMsDUEXwZzs, address:79.127.219.41:4925): protocol: read tcp 139.59.144.22:33812->79.127.219.41:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030297717 (430h35m8.04596733s) [noise=true , address=77.90.31.19:4846 , piece_id=MOMLIPX2V2BHWIU66SML35WWK6LRRDUWRUJZ6UT2HYKOA5ZWQ5IQ, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: context deadline exceeded; context deadline exceeded Mar 28 14:46:05.030604513 (430h35m8.04627423s) [noise=true , address=79.127.219.38:4818 , piece_id=YKIWBWXFSEFPB63JTXQC4BA5YIIOCZWIOKQ4GRIBXZJ3KFCZRQKA, seg_pos={0 0}, upload=413459]: Done uploading piece. err: ecclient: upload failed (node:12ZvUdXiXPFGqmVzYffAp735GjzaSMcyUfTD61Yi6us9ETg7Ath, address:79.127.219.38:4818): protocol: read tcp 139.59.144.22:58514->79.127.219.38:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 28 14:46:05.030645703 (430h35m8.046315306s) [seg_pos={0 0}, upload=413459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 28 14:46:05.141824246 (430h35m8.157493849s) [upload=413459]: Done uploading segments... Mar 28 14:46:05.141873159 (430h35m8.157542765s) [upload=413459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.600600183s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)