Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....X...................................................] duration: 2.895460768s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.89549365s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....X..................................................] duration: 2.736144289s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:18:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.736188224s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration: 2.241606429s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.241632316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration: 2.519872638s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:17:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.519899936s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration: 2.30284363s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:16:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.302876202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration: 2.851294275s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:16:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.851333198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration: 2.308199237s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:15:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.308291854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration: 2.632123181s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.632152511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 2.434978602s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:14:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.435012438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration: 2.560159636s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:14:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.560186965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 2.415444895s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:14:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.415476258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 2.840606252s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.840637976s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 2.275740867s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:13:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.275764823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....X............................................] duration: 2.891186414s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:13:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.891227734s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.................................] duration: 2.411366652s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:12:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.411391582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....X...........................................] duration: 2.891573242s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:12:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.891599559s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXX................................] duration: 2.493471384s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.493501687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....X..........................................] duration: 2.892616818s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:12:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.892635912s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXX...............................] duration: 2.338854019s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:12:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.338884502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXX..............................] duration: 2.319345871s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:11:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.319375768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXX.............................] duration: 2.296723593s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:11:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.296750384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.791801693s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:11:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.791823154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.....X.........................................] duration: 2.71504621s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:10:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.715076774s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.942720067s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:10:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.942749348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXX............................] duration: 2.310249422s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:10:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.310284752s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.561195589s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:10:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.561241946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.727583459s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:09:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.727607272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....X........................................] duration: 2.502872202s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:09:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.502951809s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 3.109048828s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:09:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.109083302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 2.387126278s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.387167053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....X.......................................] duration: 2.837385865s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:09:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.837412591s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 1.733890801s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.734914564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration: 2.411172616s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:09:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.411242006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 3.02783387s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:08:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.027877955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration: 2.347665022s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.347776728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....X......................................] duration: 2.776145432s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:08:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.776180844s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 1.710657389s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.71068376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.....X.....................................] duration: 2.868784371s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:08:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.868821307s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration: 2.25107729s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:07:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.251110468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 3.064978688s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.065009051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXX.......................] duration: 2.375474316s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.375507939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XXX........................................................] duration: 1.852191154s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:07:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.852210208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 2.607966063s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.608027521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration: 2.398567313s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:06:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.398596848s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 2.479019s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.479050723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 2.852501062s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:06:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.852553929s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration: 2.759611497s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:06:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.759655452s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 3.094300031s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:06:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.09436031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XXX.......................................................] duration: 1.685501221s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:05:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685541657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...XXX......................................................] duration: 1.572283937s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:05:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.572311982s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)