Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 3.109048828s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:09:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.109083302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 2.387126278s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.387167053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....X.......................................] duration: 2.837385865s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:09:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.837412591s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 1.733890801s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.734914564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration: 2.411172616s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:09:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.411242006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 3.02783387s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:08:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.027877955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration: 2.347665022s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.347776728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....X......................................] duration: 2.776145432s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:08:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.776180844s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 1.710657389s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.71068376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.....X.....................................] duration: 2.868784371s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:08:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.868821307s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration: 2.25107729s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:07:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.251110468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 3.064978688s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.065009051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXX.......................] duration: 2.375474316s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.375507939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XXX........................................................] duration: 1.852191154s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:07:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.852210208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 2.607966063s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.608027521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration: 2.398567313s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:06:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.398596848s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 2.479019s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.479050723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 2.852501062s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:06:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.852553929s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration: 2.759611497s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:06:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.759655452s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 3.094300031s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:06:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.09436031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XXX.......................................................] duration: 1.685501221s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:05:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685541657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...XXX......................................................] duration: 1.572283937s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:05:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.572311982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 2.919666918s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:05:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.919698947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 2.527955141s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.528011029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....X..................................] duration: 2.673680368s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:04:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.67377691s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 2.829094174s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:04:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829129052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....XXX.....................................................] duration: 1.644798307s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.644826619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 2.424302491s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:04:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.424334821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.....X.................................] duration: 2.694285616s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:03:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.694311606s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 2.920646544s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:03:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.920673117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 2.347976343s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:03:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.348012555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....XXX....................................................] duration: 1.59670539s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:03:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.596738613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 2.812793886s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:03:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.812831649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.....X................................] duration: 2.728090219s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:02:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.728108956s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.....X...............................] duration: 2.728179675s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:02:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.72826868s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......XXX...................................................] duration: 1.676015197s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:02:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.676047418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 2.295148155s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.295187246s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 2.868472434s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:02:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.868507126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.....X..............................] duration: 2.502703132s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:01:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.50272859s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 3.532138605s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532171606s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.....X.............................] duration: 2.848043985s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:01:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.848081113s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......XXX..................................................] duration: 2.219167794s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:01:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.219202885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 3.225793188s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:01:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.225821234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 2.696566504s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:01:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.696629467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 2.497916062s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-11 00:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.498027595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 2.810469444s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-11 00:00:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.810497659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........XXX.................................................] duration: 1.603397419s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-11 00:00:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.603433476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.....X............................] duration: 2.850242752s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-11 00:00:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.8503489s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 2.378687861s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:59:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.378718478s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........XXX................................................] duration: 1.667745858s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:59:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.667772292s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)