Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.462546792s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46259909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.896412156s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.896450995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.069753892s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.069793487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.864595887s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:12:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.864622703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.031569112s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.031622245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.928917156s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:11:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.928957199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002247188s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00227023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.148097912s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.148127715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.060488243s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.060519559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.374304287s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:22:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.374339125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.377740582s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:21:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.377778572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.282984623s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:20:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.283018897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.259596927s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:19:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.259621356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.091445387s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:19:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.091480627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.112139392s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:18:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.112231259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.306579538s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:17:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.306621018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.022040639s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:16:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.022072121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.105696473s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:15:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.105726988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 3.349431501s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:15:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.349462451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.241081473s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:14:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.241115098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.234767539s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:14:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.234820074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.294802984s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.294881424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 3.066018116s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.066055255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.088433181s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:11:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.088481673s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 3.340436872s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:10:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.340477619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.994871106s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:09:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.994901005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.761271784s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:09:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.761326037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.645699934s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:08:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645775267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.412364847s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:08:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.412399074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 2.063625295s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:08:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.063771422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.656723988s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:07:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.656756808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.229964526s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:07:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.23002223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.671392988s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:06:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.67141793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 3.282740791s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:06:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.282772977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.354375398s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.354533219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 2.12206856s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:05:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.122164738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 2.130224749s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:05:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.130257376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.366862742s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:05:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.366914749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.627505614s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:04:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.627546727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.057814394s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:04:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.057832534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.463664681s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:04:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.463702127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.32452463s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:03:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.324552994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 2.077347281s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:03:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.077376118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.476000277s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:02:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.476031599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.554598065s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:02:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.554631113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.280343804s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:02:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.28054869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 2.074328107s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:01:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.074355211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.415970781s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:01:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.41602123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.549049128s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 04:00:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.549084897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............................X.............................] duration: 3.373512758s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.37354443s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)