Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.908032426s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:36:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.90807639s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.704844977s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:35:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.70487627s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.719671608s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:34:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.719699906s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.79209606s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:33:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.792202979s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.947485127s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:33:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.947515516s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.789940858s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789980267s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.489126467s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:30:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.489149452s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.286030785s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:30:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.286051563s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.805882361s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:28:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.805915144s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.674042458s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:28:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.674065984s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.665243936s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:27:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.665331125s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.771630928s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:26:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.771657145s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.643845181s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:25:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.643973197s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.545755218s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:24:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.545782914s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.928755588s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.928839269s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.694302402s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:23:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.694350392s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.811973672s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:23:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.812002269s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.721583778s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:22:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.721625084s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.821178396s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:22:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.821340421s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00145156s trace: https://jaeger.dev.storj.io/trace/49d31019634df24 FAIL (2025-12-09 01:22:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001479555s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.344528494s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:07:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.344559759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.317337752s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:06:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.317434079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.501298367s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:05:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.501325176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.378798852s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:04:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378825181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.35984796s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359886307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.423229388s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:02:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.423259679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.397523293s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:02:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397608869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.43296439s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:01:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.433039214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.415036063s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 22:00:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.415061491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.432985155s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:59:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.433011584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.376080838s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:58:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376112544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.500853966s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.500875714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.400306857s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:56:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.400337691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.325329148s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:55:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.32538948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.607029666s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:54:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.607060438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.378892001s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:53:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378929851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.388249248s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388281495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.430500809s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.430525927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.455873056s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.455921409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.365680029s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.365704196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.340578495s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:49:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340609093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.425180717s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:48:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.425226803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.353478218s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:47:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.353505739s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.377616389s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:47:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.377645132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.394625219s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.394678661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.39095558s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:45:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390992537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.334210095s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:45:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.334237563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.34926469s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:44:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349293802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.3710703s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:44:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371108235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.319619529s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:43:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.319648415s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)