Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 2.378687861s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:59:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.378718478s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........XXX................................................] duration: 1.667745858s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:59:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.667772292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.....X...........................] duration: 2.660981358s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:59:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.661006808s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 3.389339472s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:59:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.389365112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 2.257545643s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:58:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.257570394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........XXX...............................................] duration: 1.564037693s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:58:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.56408404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 3.295977547s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:58:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.296002591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.....X..........................] duration: 2.770133545s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:58:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.770231631s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........XXX..............................................] duration: 1.628379196s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:58:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.628403416s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 2.517454173s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:57:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.517486871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 2.943279416s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:57:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.943308681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............XXX.............................................] duration: 1.676199333s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:57:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.676438939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration: 2.362457534s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:57:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.36248367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.....X.........................] duration: 2.470395363s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:57:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.470421106s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............XXX............................................] duration: 1.729883633s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:57:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.72992661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration: 2.344598554s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:56:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.344624909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.....X........................] duration: 2.63825967s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:56:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.638290996s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 2.884934717s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:56:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.884964342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............XXX...........................................] duration: 1.586371777s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:55:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.586413778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 2.798855647s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:55:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.798884508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 2.320786065s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:55:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.320816025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....X.......................] duration: 2.784058622s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:55:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.784087953s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............XXX..........................................] duration: 1.839069368s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:55:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.839856682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 3.378433879s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:55:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.378461105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 2.497826726s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:54:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.497856484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................XXX.........................................] duration: 1.700001536s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.700029864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....X......................] duration: 2.74949738s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:54:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.749590398s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 2.777916716s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:54:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.778022758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................XXX........................................] duration: 1.676364124s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:54:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.676392601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 2.437558435s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:54:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.437577953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................XXX.......................................] duration: 1.515666051s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:53:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.515691032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................XXX......................................] duration: 1.660266446s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:53:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.660292484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....X.....................] duration: 2.617324159s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:53:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.617351985s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................XXX.....................................] duration: 1.663727677s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:53:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.663758035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 3.384653018s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:53:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.384698595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 2.554926856s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:53:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.554959916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................XXX....................................] duration: 1.632774355s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:52:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.632854274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................XXX...................................] duration: 1.634730446s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:52:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.634757104s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.....X....................] duration: 2.58407671s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:52:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.58411802s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 2.859403695s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:52:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.859439066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 2.385233788s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:52:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.385264173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................XXX..................................] duration: 1.618280896s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:51:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.61831127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.....X...................] duration: 2.628280645s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:51:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.628309478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 2.593351555s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:51:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.593379873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 2.834586654s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:51:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.834690438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................XXX.................................] duration: 1.61112824s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:51:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.611152732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.....X..................] duration: 2.85351986s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:50:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.853552463s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 2.394678167s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:50:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.394699751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 51.898830932s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:50:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 51.89885378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 3.263453517s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:50:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.26354391s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)