Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 1.802687301s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:39:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.802750771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 3.474456899s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 03:38:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.474484882s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 1.668706701s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:38:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668733983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 1.902029275s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:38:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.902083833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 3.522475093s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 03:37:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.522510267s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 2.178338738s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:36:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.178368552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 3.398946032s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 03:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.39898091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............................X..] duration: 3.874203994s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 03:36:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.874299557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 1.849472586s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:35:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.849528122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............................X.] duration: 3.484205048s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 03:35:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.484228467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X............................X.............................X] duration: 1m0.105629745s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 03:35:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.105728953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 1.578642773s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:34:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.578671888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 1.560837657s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:34:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.560875653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X............................X.............................] duration: 4.265806845s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:33:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.265839472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X............................X............................] duration: 3.286890542s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:33:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.286964164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 1.637365062s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:33:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.637411183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X............................X...........................] duration: 3.317901015s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:33:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.317938748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 2.197045223s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:32:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.197071324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X............................X..........................] duration: 3.243348405s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.243399565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 2.155906305s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:32:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.155940569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................X.........................] duration: 4.292601767s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.292633692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 1.606464152s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:31:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.606506999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 2.069228057s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:30:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.069269539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X............................X........................] duration: 3.251018927s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:30:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.251050027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................X.......................] duration: 3.668454311s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:29:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.66848483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 2.089930397s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:29:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.089955807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.650475514s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:29:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.650494741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X............................X......................] duration: 3.544924528s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.544960561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X............................X.....................] duration: 3.114441947s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:28:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.114484049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X............................X....................] duration: 3.065839351s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:28:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.065884599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.571064017s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:28:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.571090555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........X............................X...................] duration: 3.336784601s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:28:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.336804094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 1.654388525s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:27:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65442496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........X............................X..................] duration: 3.240850009s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:27:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.240898126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 1.668763364s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:26:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668805252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.646788842s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:26:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.646830677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........X............................X.................] duration: 3.308605004s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:26:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.308631365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.844876101s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:25:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.844910092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........X............................X................] duration: 3.765031565s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:25:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.765064597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........X............................X...............] duration: 3.746916749s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:24:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.746951766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001555288s trace: https://jaeger.dev.storj.io/trace/420d42111d9ddecd FAIL (2026-04-18 03:24:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001649497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........X............................X..............] duration: 3.35084168s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:23:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.350872582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........X............................X.............] duration: 3.610580784s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:22:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.610613291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........X............................X............] duration: 3.407385651s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.407421173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........X............................X...........] duration: 3.940629072s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:20:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.940648712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........X............................X..........] duration: 3.250217964s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:19:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.250253663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........X............................X.........] duration: 4.864630973s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.864675177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........X............................X........] duration: 3.307444997s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:17:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.307545248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........X............................X.......] duration: 4.433914713s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:17:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.433945693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........X............................X......] duration: 3.075465735s trace: https://jaeger.dev.storj.io/trace/3e102f1344661b95 FAIL (2026-04-18 03:16:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Apr 18 03:11:39.783567241 (250h54m50.377678485s) [upload=241217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:11:39.783573916 (250h54m50.377685162s) [upload=241217]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:11:39.898708131 (250h54m50.492819382s) [seg_pos={0 0}, upload=241217]: Begin upload segment... Apr 18 03:11:39.898831871 (250h54m50.492943111s) [seg_pos={0 0}, upload=241217]: Done begin upload segment. Apr 18 03:11:39.898859450 (250h54m50.492970701s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.898886247 (250h54m50.492997498s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.899594583 (250h54m50.493705834s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900145699 (250h54m50.494256949s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.900923462 (250h54m50.495034712s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.901506736 (250h54m50.49561799s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.906924652 (250h54m50.501035902s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.908363341 (250h54m50.502474603s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.911853192 (250h54m50.505964456s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.917206648 (250h54m50.511317895s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.923524194 (250h54m50.517635524s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.926464962 (250h54m50.520576215s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.932864681 (250h54m50.526975932s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.933984074 (250h54m50.528095322s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.934574242 (250h54m50.528685493s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935956842 (250h54m50.53006809s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.935985842 (250h54m50.530097092s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.938083400 (250h54m50.532194651s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Uploading piece... Apr 18 03:11:39.990792921 (250h54m50.584904178s) [noise=true , address=145.79.227.34:4931 , piece_id=CDZBDINQXEQF3WX2CR7XNPFH7WY25UENQJUMJ7HAMLBPXBFA4ZZQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:145.79.227.34:4931): protocol: rpc: dial tcp 145.79.227.34:4931: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.016741317 (250h54m50.610852565s) [noise=true , address=145.79.227.49:4346 , piece_id=K6WORO2ZJD6IM7267J7KWSVF6UPECT6G2O5CCHIS4E2MPXGHY5WQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: rpc: dial tcp 145.79.227.49:4346: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.074147269 (250h54m50.668258517s) [noise=true , address=167.17.72.71:4782 , piece_id=J3KZU35WYR6BKXYKIAWFJ6UK7YB6M5325TVM42GQHFAUBILNEJQQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:39406->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:11:40.464857538 (250h54m51.058968777s) [upload=241217]: Next returned nil segment Apr 18 03:11:40.464873580 (250h54m51.05898481s) [upload=241217]: Waiting for error group managing segments... Apr 18 03:11:40.627077599 (250h54m51.22118883s) [noise=true , address=145.79.227.32:4854 , piece_id=ZKZMPRQQVG4G7J3M3YWOMLBXB4ZRQMZVQ52GBEOOROMFEQFYV2HQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.647993477 (250h54m51.242104715s) [noise=true , address=145.79.227.47:4297 , piece_id=PRH4NGOOQQCJK52G7PFPGROCZUNZQLXOSNJGI24PY6YZJM3NIOFA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.654472842 (250h54m51.248584081s) [noise=true , address=77.90.31.11:4580 , piece_id=SOUI25B7OMJYHJBJAV5CHW4T7VRTZUHV5AONJCFMBAY33DQ2DNPA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.667156588 (250h54m51.261267832s) [noise=true , address=77.90.31.14:4678 , piece_id=TB7FVQVLVQF656M6SBRVC2WTGHLDUHHS6STXLX4X4VNC2KDWVKSQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.678426254 (250h54m51.272537493s) [noise=true , address=145.79.227.21:4480 , piece_id=RRVLHT373HUZKRJQUDHW6IDPF5P4YOOXPSLAEPJJOHA7OPMMFMMA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:40.692047006 (250h54m51.286158298s) [noise=true , address=145.79.227.11:4757 , piece_id=YP7APJUGLIOOXQZDMXXN2YIOWBH5DUGWOSMTJTRXDIIZ4DSLEUHQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.166456588 (250h54m51.760567831s) [noise=true , address=193.149.164.71:4291 , piece_id=J2KXIXG74FU4A6FE3RTKZHHYERHYIVAPVM2YUILWWNXA7UZOQWAA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.224710612 (250h54m51.818821855s) [noise=true , address=79.127.205.233:4172 , piece_id=JM7IWPVVVAULWBWYWSXNHM4K3MX4HGHLHRMFKL4QZCPS7N6EL7RA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.310947060 (250h54m51.905058303s) [noise=true , address=79.127.205.235:4280 , piece_id=VJI77RZS4I3ZVPZ7QV3V55B2UL2QIVLG76SLT3JK4RKVHKVPIJXQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.344056896 (250h54m51.938168144s) [noise=true , address=79.127.205.237:4231 , piece_id=J32LPNCAONZVM3E6FJ5SPJNE3UAPOGGC3QPCUIIMXGX6FWYJMHAQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:41.826372515 (250h54m52.420483774s) [noise=true , address=145.79.227.44:4211 , piece_id=TR4DQMPMR2DWPY6E7DPH5DBOVS2K6LSZKP6NPWWZKIIPZQOKJFDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.838321211 (250h54m54.432432454s) [noise=true , address=193.149.164.143:4695 , piece_id=5JHLQPJGZUJZXSCYRPZREDT3VAZF4IUKOSLYMZUIHGSF5ID4XCOQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:11:43.916483071 (250h54m54.510594321s) [noise=true , address=77.90.31.17:4794 , piece_id=AP5W2KL273E4QNNMXV65G6ESJZD5KWEWMB4QIYOWGM3ZBUDYX4MQ, seg_pos={0 0}, upload=241217]: Done uploading piece. err: <nil> Apr 18 03:12:39.495845556 (250h55m50.089956816s) [noise=true , address=167.17.72.70:4778 , piece_id=MFD5V4A6HIZOUI4UP7EPVCILKDTNVQJIVWQV2SSVTMJYSZ6AVDDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:1DWfr9QGBeHeMGUazw9QEvSKJA2Y6q5W8hXLfV8zFCvDTScgwp, address:167.17.72.70:4778): protocol: write tcp 139.59.144.22:47950->167.17.72.70:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496111429 (250h55m50.090222686s) [noise=true , address=145.79.227.38:4993 , piece_id=OGOFIHXFJHZW5462PCK36LL6SGNPSSQX4FNBUPQCLHGUZ5FZ4SDA, seg_pos={0 0}, upload=241217]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: rpc: dial tcp 145.79.227.38:4993: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:12:39.496336055 (250h55m50.090447289s) [seg_pos={0 0}, upload=241217]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:12:39.602384040 (250h55m50.196495293s) [upload=241217]: Done uploading segments... Apr 18 03:12:39.602431411 (250h55m50.196542664s) [upload=241217]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.075530819s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)