Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 12.363168ms trace: https://jaeger.dev.storj.io/trace/40626e7a7da8f180 FAIL (2025-12-11 02:05:53 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 12.383087ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 754.945711ms trace: https://jaeger.dev.storj.io/trace/782286c78e404cd7 FAIL (2025-12-11 02:05:46 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 754.971342ms edge authservice http bgp eu1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 590.887325ms trace: https://jaeger.dev.storj.io/trace/683444a6908fe827 FAIL (2025-12-11 02:05:22 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 590.917141ms edge authservice http bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 11.791188ms trace: https://jaeger.dev.storj.io/trace/10e10589a02debf7 FAIL (2025-12-11 02:05:05 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 11.817867ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 9.363731ms trace: https://jaeger.dev.storj.io/trace/782286c78e404cd7 FAIL (2025-12-11 02:04:49 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 9.440219ms edge authservice http bgp eu1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 9.774077ms trace: https://jaeger.dev.storj.io/trace/683444a6908fe827 FAIL (2025-12-11 02:04:46 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 9.801205ms edge authservice http bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.712652902s trace: https://jaeger.dev.storj.io/trace/608bb17ec40123b0 FAIL (2025-12-11 02:04:24 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 2.712683027s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 8.143355ms trace: https://jaeger.dev.storj.io/trace/6a4ca0c2b30ce82b FAIL (2025-12-11 02:03:34 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 8.17192ms edge authservice http bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 8.361519ms trace: https://jaeger.dev.storj.io/trace/608bb17ec40123b0 FAIL (2025-12-11 02:03:22 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 8.388908ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.389786067s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:32:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.389807114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.484586376s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:32:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.484613078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.310163634s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:31:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.310212939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 2.441090586s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:30:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.44114427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 2.368833013s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:29:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.368863636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 3.028258938s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.028350551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration: 2.353313767s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:28:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.353348459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration: 2.341204062s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:27:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.341233712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration: 2.328992906s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:26:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.329022452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration: 2.547437939s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.54747894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration: 2.543425261s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:25:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.543455929s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.916216157s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:24:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.916247082s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration: 2.339426232s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:24:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.339459295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.749057505s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.749103694s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration: 2.227615714s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:23:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.227643973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration: 2.443681333s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:23:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.443701477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.767990676s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:23:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.768014879s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration: 2.438829211s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:22:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.43885787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.848735555s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.848777147s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration: 2.237960567s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.238056148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.506947719s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:21:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.506978206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration: 2.512696255s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:21:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.51272328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.554631039s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.55466406s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration: 2.359206569s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:20:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.359296972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....X.....................................................] duration: 2.753858189s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:19:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.753876828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration: 2.548232315s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.548262334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....X....................................................] duration: 2.648346587s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.648373338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX........................................] duration: 2.339955135s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:18:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.339984324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....X...................................................] duration: 2.895460768s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.89549365s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....X..................................................] duration: 2.736144289s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:18:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.736188224s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration: 2.241606429s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.241632316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration: 2.519872638s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:17:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.519899936s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration: 2.30284363s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:16:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.302876202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration: 2.851294275s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:16:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.851333198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration: 2.308199237s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:15:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.308291854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration: 2.632123181s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.632152511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 2.434978602s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:14:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.435012438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration: 2.560159636s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:14:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.560186965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 2.415444895s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:14:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.415476258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 2.840606252s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.840637976s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 2.275740867s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:13:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.275764823s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)