Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.....X.................] duration: 2.627995338s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:50:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.62801876s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................XXX................................] duration: 1.58784155s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:50:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.587875268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.....X................] duration: 2.68053542s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:50:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.680652698s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................XXX.................................] duration: 2.828013327s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:49:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.828037734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.108389443s trace: https://jaeger.dev.storj.io/trace/4a2dd5aa8d8fad7a FAIL (2025-12-10 23:49:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 226541773s) [seg_pos={0 0}, upload=112810]: Begin upload segment... Dec 10 23:48:58.246287283 (128h20m38.22663992s) [seg_pos={0 0}, upload=112810]: Done begin upload segment. Dec 10 23:48:58.246345703 (128h20m38.226698343s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.246360740 (128h20m38.226713388s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247083946 (128h20m38.227436588s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.247692343 (128h20m38.228044989s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.248154046 (128h20m38.228506691s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250015645 (128h20m38.230368286s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250052525 (128h20m38.230405172s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250367231 (128h20m38.230719874s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250526599 (128h20m38.230879246s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250675248 (128h20m38.231027908s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.250925440 (128h20m38.231278083s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251060530 (128h20m38.231413174s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251373151 (128h20m38.231725793s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.251857728 (128h20m38.232210374s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.252339639 (128h20m38.232692282s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.253463443 (128h20m38.23381609s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254018244 (128h20m38.234370885s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.254521300 (128h20m38.234873942s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Uploading piece... Dec 10 23:48:58.801841681 (128h20m38.782194318s) [upload=112810]: Next returned nil segment Dec 10 23:48:58.801855452 (128h20m38.782208089s) [upload=112810]: Waiting for error group managing segments... Dec 10 23:48:58.958875956 (128h20m38.939228593s) [noise=true , address=45.91.116.12:4259 , piece_id=TED2FLFAAHPHRLQKXP5U56KKWCZLOJSULRKKPSK3BAIYKHC3LLWQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960230967 (128h20m38.940583604s) [noise=true , address=45.91.116.9:4288 , piece_id=UPDK3342HOEGXTNQHDCG7MT6NJFSKBFD6UASAYUNTDGDJL4F7T2A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.960382910 (128h20m38.940735605s) [noise=true , address=145.79.227.12:4779 , piece_id=6U2Z3RXTUSXQRUDP6RMLROH5PAXW6KPN66MT7VITV5Y7H6IDJIRQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.967583959 (128h20m38.947936601s) [noise=true , address=145.79.227.29:4973 , piece_id=L5KOV5JBNP5GURN4UQFK4O3ET47W22BXHMCISSRO4C6OXXZR3XUQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.969251187 (128h20m38.949603824s) [noise=true , address=145.79.227.33:4886 , piece_id=ZITLXZOHGZD5KZDLG4DIRF7BP7NZJCRCPTRUP6PGTX2SPDMSJTSA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.974549496 (128h20m38.954902133s) [noise=true , address=145.79.227.45:4245 , piece_id=N5Z6YLVH6AZ675D5TYUZLQK5W6YL45X5FQL4LTZQWP55TTIVPHIQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.975160528 (128h20m38.955513165s) [noise=true , address=45.91.116.13:4181 , piece_id=NHMXGY4ADRYGULC44K4G2PUBW3NUGHETUPEP77ZAKYKYCZQ46GKQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.982414116 (128h20m38.962766752s) [noise=true , address=145.79.227.5:4244 , piece_id=SONCZF3YE3UWVSBP7YBVYEQPWIDBFZKYIFSF66A2QWGHWL3J47ZA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:58.987979940 (128h20m38.968332577s) [noise=true , address=45.91.116.4:4971 , piece_id=7KJNKTHNBWC2NBO7RFAYGHV2UUESEEEZE6H7TOU7NNB2EN3GZ3JQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.048070692 (128h20m39.028423333s) [noise=true , address=45.91.116.10:4108 , piece_id=ZFECTXMWQ4ST2FFD3LHZU7URBWNQNF5WHMPAIFMBPFLHZKX4WIEQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:48:59.349464001 (128h20m39.329816637s) [noise=true , address=145.79.227.23:4560 , piece_id=GAA4HNS45K6QH3P3TX2HIE6VMJDZHI4RWPVNM65HAASLXE5CUT3Q, seg_pos={0 0}, upload=112810]: Done uploading piece. err: <nil> Dec 10 23:49:57.831765247 (128h21m37.812117892s) [noise=true , address=77.90.31.13:4636 , piece_id=ZKIPS3DB2PYA52K7ABDYP744YYZ7RPIOI6SIRA4KTCSC43RCFHOQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831829354 (128h21m37.812182s) [noise=true , address=77.90.31.12:4595 , piece_id=XS2HHSLJY3A2ACU45UPGAPFAKNF33FXGJ4HG3CNGG64ALL5ASN3A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: rpc: dial tcp 77.90.31.12:4595: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.831951926 (128h21m37.812304568s) [noise=true , address=77.90.31.18:4834 , piece_id=4VAL6G7LS3BEKVQ6KYN25FEMW3QA6X4IQ6ZRMPURHVBJMGW2PIWA, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832095043 (128h21m37.812447688s) [noise=true , address=77.90.31.16:4767 , piece_id=OU6MARMQ52HJMVN4YIQUMEPGZVNHACUDAPR6ADOOZJWO42JGMDVQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832249534 (128h21m37.812602174s) [noise=true , address=77.90.31.15:4702 , piece_id=XXAMPW5B4QTK5NHFJ4JLUVAYP4UD52EH2UB4JC4BVXCR6LWPZF6A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42630->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832269992 (128h21m37.812622632s) [noise=true , address=77.90.31.16:4745 , piece_id=GFKY7673KADSZENZO4BFOBJ254OB7OUDBEREUBSKYGVUWDP77L7A, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:49:57.832281948 (128h21m37.812634572s) [noise=true , address=77.90.31.11:4588 , piece_id=NUMOBQ7XUFQF4QRN7UBU3GUNV2QG7LNKIJFL6PPB5KW4OFD6CDBQ, seg_pos={0 0}, upload=112810]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:57.832305298 (128h21m37.812657921s) [seg_pos={0 0}, upload=112810]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:57.938807536 (128h21m37.91916016s) [upload=112810]: Done uploading segments... Dec 10 23:49:57.938838615 (128h21m37.919191251s) [upload=112810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.108409916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................XXX................................] duration: 2.827799995s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:49:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.827824524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.....X...............] duration: 2.761781234s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:49:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.761805119s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................XXX...............................] duration: 1.576585745s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:49:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.576655612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.112042533s trace: https://jaeger.dev.storj.io/trace/63cf6c5ff045109a FAIL (2025-12-10 23:48:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== VO7FVWSLMXOZVUU26XQI2CUHWLQA5GDHNIQPXNT2LCLSQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.477900249 (128h19m29.458252886s) [noise=true , address=77.90.31.14:4673 , piece_id=ZAKPHONELEU4MLYFH62XYR2NLCKASADH7EU4AD6DRWWAZEOKGE5A, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.482032591 (128h19m29.462385217s) [noise=true , address=5.231.73.2:4030 , piece_id=GWNUOTJWUKA2BIQJ7H3NDYOPFV6CHSSJ3WUTD2KWXZIY7RRLHNRA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.482510863 (128h19m29.462863492s) [noise=true , address=145.79.227.26:4678 , piece_id=BEHVDBJR6T2PLQKVREXFYR4KP2HMPB2P6KROI5JGY6VYOJQ3U6FQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.483973154 (128h19m29.464325909s) [noise=true , address=77.90.31.16:4742 , piece_id=PRQPQ37LYUSDNK5PQREIDE5YETK4J7GSLDMCC7J4D5XFBS6FKZSQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.486940105 (128h19m29.467292735s) [noise=true , address=45.91.116.10:4092 , piece_id=YSSPLZJAGNGP3WOMYPWJBLN6GO2RDHFW5TXL774HNMXR4XLKOKSQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.489191514 (128h19m29.469544143s) [noise=true , address=77.90.31.15:4721 , piece_id=HXY3TGBDWAPHG6IZXXWNC5FTESDC6NFRGCGSYQ66PC6YD7LEZNHA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.490506584 (128h19m29.470859213s) [noise=true , address=64.49.13.169:4707 , piece_id=E5XCE3P3GKA72TLERFNMEUY4WDDB2IEXBYWKBJ73EGCI5QLL4O3A, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.491177671 (128h19m29.4715303s) [noise=true , address=77.90.31.17:4783 , piece_id=2BINW5DEDY4VQTOZMEZJMFK5IZ6UNTSZDW67MOGRTQDYRVRJY7JA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.492640666 (128h19m29.472993295s) [noise=true , address=145.79.227.33:4892 , piece_id=Q3BZZWRFNGJRRNHGDFGW2M35KANMV3UZUW432AFRJSKP7WD274GQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.492849283 (128h19m29.473201906s) [noise=true , address=77.90.31.11:4560 , piece_id=5344W2F3R6PBAZFPJMOB2YLFTJZUHC54RCKE3RV7NEJ4EPPCL7RA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.494175893 (128h19m29.474528516s) [noise=true , address=45.91.116.7:4042 , piece_id=VR7AIU5GNRWZQQ6GUAQVFL4ZWOVOWQS2RF2JOX3CJO4F5XLAD25Q, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.494988096 (128h19m29.475340726s) [noise=true , address=77.90.31.17:4798 , piece_id=U2GVQ4VZ3VXP6KZJRXKMZCBZQUTH4NIWNI4AUN4NFKWCBBHD3FQA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.495827919 (128h19m29.476180548s) [noise=true , address=77.90.31.10:4543 , piece_id=ZBSRHYZ2Y6YPRA5R4MGF6KGCVW4GKB7BJ2UY62KUWMQGWWN52PXA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.498362967 (128h19m29.478715599s) [noise=true , address=145.79.227.23:4548 , piece_id=EU2SU4CKMKPOYBWO7JNOKZNWKHCFYUZX2CRSFBPVCR4TQ7WWM7KQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.500097721 (128h19m29.480450351s) [noise=true , address=77.90.31.21:4883 , piece_id=ZQPPK6YIXXDD5XI4OXE6CSXHKZPQMA3QIWI32S7NS73LFH4GOGRQ, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:49.502667389 (128h19m29.483020018s) [noise=true , address=5.231.73.4:4025 , piece_id=5WAOIGWKAC7UGLFPJ3EH74Z4TNHDA2JWWUQGDFSFF45TB5IHKACA, seg_pos={0 0}, upload=112795]: Uploading piece... Dec 10 23:47:50.059813494 (128h19m30.040166135s) [upload=112795]: Next returned nil segment Dec 10 23:47:50.059834157 (128h19m30.040186803s) [upload=112795]: Waiting for error group managing segments... Dec 10 23:47:50.233076582 (128h19m30.21342924s) [noise=true , address=145.79.227.23:4548 , piece_id=EU2SU4CKMKPOYBWO7JNOKZNWKHCFYUZX2CRSFBPVCR4TQ7WWM7KQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:50.242844007 (128h19m30.223196633s) [noise=true , address=145.79.227.33:4892 , piece_id=Q3BZZWRFNGJRRNHGDFGW2M35KANMV3UZUW432AFRJSKP7WD274GQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:50.260046120 (128h19m30.240398749s) [noise=true , address=45.91.116.7:4042 , piece_id=VR7AIU5GNRWZQQ6GUAQVFL4ZWOVOWQS2RF2JOX3CJO4F5XLAD25Q, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:50.270101365 (128h19m30.250453994s) [noise=true , address=145.79.227.26:4678 , piece_id=BEHVDBJR6T2PLQKVREXFYR4KP2HMPB2P6KROI5JGY6VYOJQ3U6FQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:50.276686816 (128h19m30.257039462s) [noise=true , address=45.91.116.10:4092 , piece_id=YSSPLZJAGNGP3WOMYPWJBLN6GO2RDHFW5TXL774HNMXR4XLKOKSQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:50.387765364 (128h19m30.368118s) [noise=true , address=5.231.73.2:4030 , piece_id=GWNUOTJWUKA2BIQJ7H3NDYOPFV6CHSSJ3WUTD2KWXZIY7RRLHNRA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:50.434297169 (128h19m30.414649811s) [noise=true , address=5.231.73.4:4025 , piece_id=5WAOIGWKAC7UGLFPJ3EH74Z4TNHDA2JWWUQGDFSFF45TB5IHKACA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:47:51.162112216 (128h19m31.142464861s) [noise=true , address=145.79.227.16:4929 , piece_id=LSE7URTEANDUZBHOX7MLX5SE53BRLWOHQRCL2PQ6NQMBXKBY5WHA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:48:02.224665832 (128h19m42.205018474s) [noise=true , address=64.49.13.169:4707 , piece_id=E5XCE3P3GKA72TLERFNMEUY4WDDB2IEXBYWKBJ73EGCI5QLL4O3A, seg_pos={0 0}, upload=112795]: Done uploading piece. err: <nil> Dec 10 23:48:49.070382127 (128h20m29.050734772s) [noise=true , address=77.90.31.10:4543 , piece_id=ZBSRHYZ2Y6YPRA5R4MGF6KGCVW4GKB7BJ2UY62KUWMQGWWN52PXA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.070560346 (128h20m29.050912991s) [noise=true , address=77.90.31.16:4742 , piece_id=PRQPQ37LYUSDNK5PQREIDE5YETK4J7GSLDMCC7J4D5XFBS6FKZSQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:1me3JXQFLGyr6REuWPk7f7arC5d72HcDbexXuez45dYBKqVA3k, address:77.90.31.16:4742): protocol: rpc: dial tcp 77.90.31.16:4742: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.070684770 (128h20m29.051037415s) [noise=true , address=77.90.31.15:4721 , piece_id=HXY3TGBDWAPHG6IZXXWNC5FTESDC6NFRGCGSYQ66PC6YD7LEZNHA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn, address:77.90.31.15:4721): protocol: rpc: dial tcp 77.90.31.15:4721: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.070794706 (128h20m29.051147348s) [noise=true , address=77.90.31.17:4798 , piece_id=U2GVQ4VZ3VXP6KZJRXKMZCBZQUTH4NIWNI4AUN4NFKWCBBHD3FQA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 139.59.144.22:59058->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.070944740 (128h20m29.051297378s) [noise=true , address=77.90.31.21:4883 , piece_id=ZQPPK6YIXXDD5XI4OXE6CSXHKZPQMA3QIWI32S7NS73LFH4GOGRQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, address:77.90.31.21:4883): protocol: rpc: dial tcp 77.90.31.21:4883: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.071283971 (128h20m29.051636616s) [noise=true , address=77.90.31.11:4560 , piece_id=5344W2F3R6PBAZFPJMOB2YLFTJZUHC54RCKE3RV7NEJ4EPPCL7RA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.071384151 (128h20m29.051736793s) [noise=true , address=77.90.31.17:4783 , piece_id=2BINW5DEDY4VQTOZMEZJMFK5IZ6UNTSZDW67MOGRTQDYRVRJY7JA, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: rpc: dial tcp 77.90.31.17:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.071461258 (128h20m29.051813899s) [noise=true , address=77.90.31.12:4618 , piece_id=IXKWYIXVO7FVWSLMXOZVUU26XQI2CUHWLQA5GDHNIQPXNT2LCLSQ, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, address:77.90.31.12:4618): protocol: rpc: dial tcp 77.90.31.12:4618: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.071615068 (128h20m29.051967714s) [noise=true , address=77.90.31.14:4673 , piece_id=ZAKPHONELEU4MLYFH62XYR2NLCKASADH7EU4AD6DRWWAZEOKGE5A, seg_pos={0 0}, upload=112795]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:49.071634661 (128h20m29.051987283s) [seg_pos={0 0}, upload=112795]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Dec 10 23:48:49.180426131 (128h20m29.160778767s) [upload=112795]: Done uploading segments... Dec 10 23:48:49.180459341 (128h20m29.160811978s) [upload=112795]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.112058379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 2.872340524s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:48:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.87236669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................XXX..............................] duration: 1.679927499s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:48:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.679967918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.....X..............] duration: 2.617104064s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:48:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.617126693s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 2.828549173s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:48:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.828576881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.103383147s trace: https://jaeger.dev.storj.io/trace/57fbb600e7736044 FAIL (2025-12-10 23:47:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== :46:49.255943547 (128h18m29.236296189s) [noise=true , address=145.79.227.28:4757 , piece_id=NS3PSMJWFFBXT7TA24PPXNBO4BE2BBOE3OSBPPCTOV2EULFG32VA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.256071740 (128h18m29.236424397s) [noise=true , address=5.231.73.3:4033 , piece_id=X6SXGGTNUAI4H44W67NLP7Y2OBEFZ3A5S3JKOVH7AXCV6HZJEY5Q, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.257111567 (128h18m29.237464214s) [noise=true , address=77.90.31.12:4614 , piece_id=WY4CDJCHN7S57JQICJIYNAYQEAUYLYVBORC4YJUAU3FGQO34JEYQ, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.257813744 (128h18m29.238166391s) [noise=true , address=145.79.227.13:4831 , piece_id=2U7TK327OIWXIYLYIXSSAAZJG2UJ2A3GH5QAA2XTNJ76ETSJL6NQ, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.257826790 (128h18m29.238179436s) [noise=true , address=45.91.116.5:4308 , piece_id=WGJWFHLSSX2C5HQ6PSOH5W3XLURYPARJOFX25FWMKAYRF5D2WG6A, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.258282017 (128h18m29.238634655s) [noise=true , address=145.79.227.11:4756 , piece_id=GHCXYNM6XO7VF3VCARKOWYGI44HIYJRMTIXCQV7257DZQOVGSFSQ, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.258688911 (128h18m29.239041555s) [noise=true , address=77.90.31.15:4720 , piece_id=7H35JMSYX7LWJ6GCMQA3BD3MOIN7L4PFEJTP42HITFVFQAIXR3SA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.259085591 (128h18m29.239438219s) [noise=true , address=145.79.227.16:4918 , piece_id=ZH2HDJP6U5556V5THFLCFCQKHY3ZVBBNUMQGJUDBPQVYRQDCKGAQ, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.259512085 (128h18m29.239864732s) [noise=true , address=77.90.31.11:4583 , piece_id=BHBUPPMTXWX6C2UE6SORUDUGF5K2XO4CQKCSHYMX4MX2JDMXZJ4Q, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.259973856 (128h18m29.2403265s) [noise=true , address=145.79.227.29:4969 , piece_id=EE7UO3PCIOYP6VD2AFKRZGRVFQTBOCYCX3XCV7DMKFF6AWUDZZSQ, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.260454855 (128h18m29.240807498s) [noise=true , address=45.91.116.14:4207 , piece_id=3A6HVVTHNV3YY4LKBFM4Z6H5XQEOASPWI3TOPZKUBEUHIGMDFP4Q, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.260949734 (128h18m29.241302358s) [noise=true , address=77.90.31.19:4875 , piece_id=KBAEHRZYWC6LBQAV75IL4ENFV7DGCBOS4D5YNAJEVAWVAT5RJTKA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.263099263 (128h18m29.243451908s) [noise=true , address=77.90.31.19:4854 , piece_id=JQ2ZDIGAHX5AQRXET2LMQKEGTOTF27XXGYNKH2J26X7I4IED6BLA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.263741627 (128h18m29.244094274s) [noise=true , address=77.90.31.18:4840 , piece_id=J3RT72J4E27B7PS7IKQZJAAEIGPOYXXJB6AJ64YA2ZXNMKHML3BA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.264354515 (128h18m29.244707162s) [noise=true , address=77.90.31.16:4763 , piece_id=BK6IR5P2KQ3XQZF22QYOOM76QG3GQ6EBKE5YGB64NPMPS2INLDRA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.264921167 (128h18m29.24527379s) [noise=true , address=45.91.116.8:4058 , piece_id=MQW24OW626TU2SJQGY66OGVPNACUEQUE7HEJTNT47TQAVPNU5LSA, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.267662465 (128h18m29.248015112s) [noise=true , address=77.90.31.10:4541 , piece_id=M5XXGB7RVNYRYU7YJLCMZMPQRGGTSV546JPBRCBGGWNLLWMI3TNQ, seg_pos={0 0}, upload=112783]: Uploading piece... Dec 10 23:46:49.790689008 (128h18m29.771041632s) [upload=112783]: Next returned nil segment Dec 10 23:46:49.790709232 (128h18m29.771061868s) [upload=112783]: Waiting for error group managing segments... Dec 10 23:46:49.965438083 (128h18m29.94579072s) [noise=true , address=145.79.227.28:4757 , piece_id=NS3PSMJWFFBXT7TA24PPXNBO4BE2BBOE3OSBPPCTOV2EULFG32VA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:49.977579518 (128h18m29.957932163s) [noise=true , address=145.79.227.13:4831 , piece_id=2U7TK327OIWXIYLYIXSSAAZJG2UJ2A3GH5QAA2XTNJ76ETSJL6NQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:49.982221602 (128h18m29.962574243s) [noise=true , address=45.91.116.14:4207 , piece_id=3A6HVVTHNV3YY4LKBFM4Z6H5XQEOASPWI3TOPZKUBEUHIGMDFP4Q, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:49.994173233 (128h18m29.974525873s) [noise=true , address=145.79.227.29:4969 , piece_id=EE7UO3PCIOYP6VD2AFKRZGRVFQTBOCYCX3XCV7DMKFF6AWUDZZSQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:49.994231552 (128h18m29.974584188s) [noise=true , address=45.91.116.8:4058 , piece_id=MQW24OW626TU2SJQGY66OGVPNACUEQUE7HEJTNT47TQAVPNU5LSA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:50.002467362 (128h18m29.982820013s) [noise=true , address=145.79.227.11:4756 , piece_id=GHCXYNM6XO7VF3VCARKOWYGI44HIYJRMTIXCQV7257DZQOVGSFSQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:50.005321148 (128h18m29.985673806s) [noise=true , address=45.91.116.5:4308 , piece_id=WGJWFHLSSX2C5HQ6PSOH5W3XLURYPARJOFX25FWMKAYRF5D2WG6A, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:50.020724935 (128h18m30.001077583s) [noise=true , address=145.79.227.34:4935 , piece_id=RY5IAEFCFZDQTNWLTSQCGPL4W4EEKCUPK4SO2OIKJUAOPVVYHJMQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:50.161979821 (128h18m30.142332457s) [noise=true , address=5.231.73.3:4033 , piece_id=X6SXGGTNUAI4H44W67NLP7Y2OBEFZ3A5S3JKOVH7AXCV6HZJEY5Q, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:46:50.881962476 (128h18m30.862315113s) [noise=true , address=145.79.227.16:4918 , piece_id=ZH2HDJP6U5556V5THFLCFCQKHY3ZVBBNUMQGJUDBPQVYRQDCKGAQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: <nil> Dec 10 23:47:48.840329654 (128h19m28.8206823s) [noise=true , address=77.90.31.19:4875 , piece_id=KBAEHRZYWC6LBQAV75IL4ENFV7DGCBOS4D5YNAJEVAWVAT5RJTKA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: rpc: dial tcp 77.90.31.19:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.840451298 (128h19m28.82080394s) [noise=true , address=77.90.31.19:4854 , piece_id=JQ2ZDIGAHX5AQRXET2LMQKEGTOTF27XXGYNKH2J26X7I4IED6BLA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE, address:77.90.31.19:4854): protocol: write tcp 139.59.144.22:52086->77.90.31.19:4854: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.840584495 (128h19m28.820937144s) [noise=true , address=77.90.31.12:4614 , piece_id=WY4CDJCHN7S57JQICJIYNAYQEAUYLYVBORC4YJUAU3FGQO34JEYQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.840674512 (128h19m28.821027153s) [noise=true , address=77.90.31.16:4763 , piece_id=BK6IR5P2KQ3XQZF22QYOOM76QG3GQ6EBKE5YGB64NPMPS2INLDRA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: write tcp 139.59.144.22:43044->77.90.31.16:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.840824435 (128h19m28.821177083s) [noise=true , address=77.90.31.11:4583 , piece_id=BHBUPPMTXWX6C2UE6SORUDUGF5K2XO4CQKCSHYMX4MX2JDMXZJ4Q, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t, address:77.90.31.11:4583): protocol: rpc: dial tcp 77.90.31.11:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.840888762 (128h19m28.821241402s) [noise=true , address=77.90.31.10:4541 , piece_id=M5XXGB7RVNYRYU7YJLCMZMPQRGGTSV546JPBRCBGGWNLLWMI3TNQ, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 139.59.144.22:38188->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.841213765 (128h19m28.821566411s) [noise=true , address=77.90.31.18:4840 , piece_id=J3RT72J4E27B7PS7IKQZJAAEIGPOYXXJB6AJ64YA2ZXNMKHML3BA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.841329986 (128h19m28.821682629s) [noise=true , address=77.90.31.15:4720 , piece_id=7H35JMSYX7LWJ6GCMQA3BD3MOIN7L4PFEJTP42HITFVFQAIXR3SA, seg_pos={0 0}, upload=112783]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 139.59.144.22:34016->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:48.841374410 (128h19m28.821727037s) [seg_pos={0 0}, upload=112783]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:47:48.941665233 (128h19m28.922017858s) [upload=112783]: Done uploading segments... Dec 10 23:47:48.941697062 (128h19m28.922049704s) [upload=112783]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.103397223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 2.841393218s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:47:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.841435835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................XXX.............................] duration: 1.645219472s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:47:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645247269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 2.956592368s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:47:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.95668153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.....X.............] duration: 2.625292443s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:46:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.625322474s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 2.926518641s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:46:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.926593398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.1111449s trace: https://jaeger.dev.storj.io/trace/521f24bd646e01d0 FAIL (2025-12-10 23:46:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ]: Begin upload segment... Dec 10 23:45:48.986750116 (128h17m28.967102753s) [seg_pos={0 0}, upload=112768]: Done begin upload segment. Dec 10 23:45:48.986789518 (128h17m28.967142154s) [noise=true , address=145.79.227.13:4831 , piece_id=6UU57UVU5DFPWMD7J2DFKKQA2RG7ARXAZDYXVQVDVX3LREOF3HJA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.987651665 (128h17m28.96800431s) [noise=true , address=77.90.31.12:4600 , piece_id=6QGN2HJ7HK543K7SDOJ5UFCM5M3LNSREVFZIML3DQRC6NS5GUNHA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.988018008 (128h17m28.96837065s) [noise=true , address=77.90.31.11:4583 , piece_id=XPNRU3LR4T4H22MD43SCEL3C22AT3W6A64YUPYE654TOX7QVDUBA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.988363836 (128h17m28.968716475s) [noise=true , address=77.90.31.19:4867 , piece_id=2ITDRJI4KXT44VBOZCMQ4TTCENARQQIKUUPP45I4PQRWJI2ZZ6LA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.988705001 (128h17m28.969057641s) [noise=true , address=45.91.116.14:4211 , piece_id=HT5IOR7XMN2ZTJPLMHK2QSDNGM34NVIAUY3SEGALVU3FTH5IWYXA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.989176393 (128h17m28.969529039s) [noise=true , address=77.90.31.18:4811 , piece_id=DRBHFC3OP62OZANZ7RZ3KJFU2YTP7KIATJ3ANBXBVHPT46L5LFUA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.989587840 (128h17m28.969940478s) [noise=true , address=5.231.73.5:4004 , piece_id=3Q7S27AAYVGWY4XSYSUIZBCQIPMV37E6ZTLSJK5EGTFHG5EQEVOQ, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.990043597 (128h17m28.970396227s) [noise=true , address=145.79.227.35:4501 , piece_id=JKTIWNR7GDRDDJXKPECE5RU26AI7BFWURKZHBK7LPOJYKV4EIDRQ, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.990480525 (128h17m28.970833165s) [noise=true , address=77.90.31.12:4615 , piece_id=OB6APDZ4F563M4XKOLWWD7VKTKNX7ZLOBZQ6G5VUPQ6HDSSY2GOA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.990928440 (128h17m28.971281082s) [noise=true , address=45.91.116.9:4289 , piece_id=FAZHJNROVSRSAPESK7ED4FIA6KHNIHXRJKCCTP2S3OBHG3P6I56A, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.991350554 (128h17m28.971703191s) [noise=true , address=145.79.227.32:4863 , piece_id=33GQR23XPPA6YYK3CLQ2IPPU5AS2ZMX7K6OFATCUB6C5WKRWKIZA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.991776064 (128h17m28.972128739s) [noise=true , address=64.49.13.137:4500 , piece_id=HJPOPMYQVGGAE3L3IEZDHCUK73WJIYSD2PPN33IASU5KDK2ZT7KA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.992145478 (128h17m28.972498113s) [noise=true , address=64.49.13.169:4707 , piece_id=2OF7QZ3TZ6C63NN2NYZSNQ3WJDM7QIJEBFIRRQLOACLRD7A4GZ3Q, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.993079337 (128h17m28.973431976s) [noise=true , address=145.79.227.49:4348 , piece_id=IE3OHQKAX4TH6GP2GFKTNKQX4RGUOXUVVFW5XFTNSDS5ABIXPQYQ, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.993616166 (128h17m28.973968804s) [noise=true , address=77.90.31.13:4652 , piece_id=4YYENCLYSMT6PY253RHVS2W3DXGBFGYVY4E6DLLD6BEG7DNRZ2RQ, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.994206197 (128h17m28.974558842s) [noise=true , address=145.79.227.44:4218 , piece_id=PQA7KOVQL2KKWABPQY2K7E3EBPSCW6J7B6FCUF3C7KDEZMOOBNCA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.994671849 (128h17m28.975024485s) [noise=true , address=77.90.31.15:4722 , piece_id=3YJKKDSOFQXC2XM52B7P4YDFJ56EMZDWRU2AXXF6IQJ2GK3S2IXA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:48.995186450 (128h17m28.97553909s) [noise=true , address=45.91.116.8:4071 , piece_id=4PM2LRRO3MYLHJZHBZVYMESAALN46TG5G4XIU6IKZNNOLQCKYSNA, seg_pos={0 0}, upload=112768]: Uploading piece... Dec 10 23:45:49.567822099 (128h17m29.548174756s) [upload=112768]: Next returned nil segment Dec 10 23:45:49.567848557 (128h17m29.548201205s) [upload=112768]: Waiting for error group managing segments... Dec 10 23:45:49.704699642 (128h17m29.68505228s) [noise=true , address=45.91.116.14:4211 , piece_id=HT5IOR7XMN2ZTJPLMHK2QSDNGM34NVIAUY3SEGALVU3FTH5IWYXA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.713435786 (128h17m29.693788433s) [noise=true , address=45.91.116.9:4289 , piece_id=FAZHJNROVSRSAPESK7ED4FIA6KHNIHXRJKCCTP2S3OBHG3P6I56A, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.714478026 (128h17m29.694830674s) [noise=true , address=145.79.227.49:4348 , piece_id=IE3OHQKAX4TH6GP2GFKTNKQX4RGUOXUVVFW5XFTNSDS5ABIXPQYQ, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.720056893 (128h17m29.700409534s) [noise=true , address=45.91.116.8:4071 , piece_id=4PM2LRRO3MYLHJZHBZVYMESAALN46TG5G4XIU6IKZNNOLQCKYSNA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.723862974 (128h17m29.704215611s) [noise=true , address=145.79.227.32:4863 , piece_id=33GQR23XPPA6YYK3CLQ2IPPU5AS2ZMX7K6OFATCUB6C5WKRWKIZA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.725337329 (128h17m29.705689966s) [noise=true , address=145.79.227.13:4831 , piece_id=6UU57UVU5DFPWMD7J2DFKKQA2RG7ARXAZDYXVQVDVX3LREOF3HJA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.805268180 (128h17m29.785620816s) [noise=true , address=145.79.227.35:4501 , piece_id=JKTIWNR7GDRDDJXKPECE5RU26AI7BFWURKZHBK7LPOJYKV4EIDRQ, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:49.886225470 (128h17m29.866578125s) [noise=true , address=5.231.73.5:4004 , piece_id=3Q7S27AAYVGWY4XSYSUIZBCQIPMV37E6ZTLSJK5EGTFHG5EQEVOQ, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:50.375706351 (128h17m30.356058999s) [noise=true , address=64.49.13.137:4500 , piece_id=HJPOPMYQVGGAE3L3IEZDHCUK73WJIYSD2PPN33IASU5KDK2ZT7KA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:45:50.606824803 (128h17m30.587177442s) [noise=true , address=145.79.227.44:4218 , piece_id=PQA7KOVQL2KKWABPQY2K7E3EBPSCW6J7B6FCUF3C7KDEZMOOBNCA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:46:00.681478194 (128h17m40.661830835s) [noise=true , address=64.49.13.169:4707 , piece_id=2OF7QZ3TZ6C63NN2NYZSNQ3WJDM7QIJEBFIRRQLOACLRD7A4GZ3Q, seg_pos={0 0}, upload=112768]: Done uploading piece. err: <nil> Dec 10 23:46:48.588477063 (128h18m28.568829709s) [noise=true , address=77.90.31.12:4615 , piece_id=OB6APDZ4F563M4XKOLWWD7VKTKNX7ZLOBZQ6G5VUPQ6HDSSY2GOA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: rpc: dial tcp 77.90.31.12:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:48.588544751 (128h18m28.568897398s) [noise=true , address=77.90.31.15:4722 , piece_id=3YJKKDSOFQXC2XM52B7P4YDFJ56EMZDWRU2AXXF6IQJ2GK3S2IXA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:48.588844226 (128h18m28.569196868s) [noise=true , address=77.90.31.12:4600 , piece_id=6QGN2HJ7HK543K7SDOJ5UFCM5M3LNSREVFZIML3DQRC6NS5GUNHA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:48.588875032 (128h18m28.569227676s) [noise=true , address=77.90.31.18:4811 , piece_id=DRBHFC3OP62OZANZ7RZ3KJFU2YTP7KIATJ3ANBXBVHPT46L5LFUA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6, address:77.90.31.18:4811): protocol: rpc: dial tcp 77.90.31.18:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:48.589418220 (128h18m28.569770862s) [noise=true , address=77.90.31.13:4652 , piece_id=4YYENCLYSMT6PY253RHVS2W3DXGBFGYVY4E6DLLD6BEG7DNRZ2RQ, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: write tcp 139.59.144.22:33562->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:48.590045712 (128h18m28.570398354s) [noise=true , address=77.90.31.11:4583 , piece_id=XPNRU3LR4T4H22MD43SCEL3C22AT3W6A64YUPYE654TOX7QVDUBA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t, address:77.90.31.11:4583): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:46:48.590114252 (128h18m28.570466882s) [noise=true , address=77.90.31.19:4867 , piece_id=2ITDRJI4KXT44VBOZCMQ4TTCENARQQIKUUPP45I4PQRWJI2ZZ6LA, seg_pos={0 0}, upload=112768]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 139.59.144.22:48064->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:48.590138534 (128h18m28.570491164s) [seg_pos={0 0}, upload=112768]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:46:48.696899674 (128h18m28.677252311s) [upload=112768]: Done uploading segments... Dec 10 23:46:48.697046008 (128h18m28.677398645s) [upload=112768]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.111170845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................XXX............................] duration: 1.627857339s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:46:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.62788279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.....X............] duration: 2.861582677s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:46:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.861621623s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 2.720997318s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:45:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.721010354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.117894239s trace: https://jaeger.dev.storj.io/trace/357d110fca819a45 FAIL (2025-12-10 23:45:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== : Begin upload segment... Dec 10 23:44:45.002561982 (128h16m24.98291462s) [seg_pos={0 0}, upload=112754]: Done begin upload segment. Dec 10 23:44:45.002611320 (128h16m24.982963966s) [noise=true , address=77.90.31.15:4731 , piece_id=3CWTL4NSPS2ONXEEZ6AXE5UYOXWBLYEUP255WJQ3HYVULNMEF5AA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.002717015 (128h16m24.983069653s) [noise=true , address=77.90.31.16:4736 , piece_id=4A4DTFFBSNMAV5PVWEVJQ4JHAOM2KBWSADAHYT3AY2LUX5VEAWEA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.003487249 (128h16m24.983839894s) [noise=true , address=77.90.31.12:4619 , piece_id=LXJA4BFZO5XSCQIZIKLQCC4WBZDW4J36MPPNP6UPLOPZPVCG4DTA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.003880428 (128h16m24.984233101s) [noise=true , address=145.79.227.11:4760 , piece_id=TWP2PR3E2YAXQH5BX7W374VFX26CW26DDL6G4NHNGOANGN7EENOA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.004145268 (128h16m24.98449792s) [noise=true , address=77.90.31.21:4900 , piece_id=HQCR3WX5ULCE67W6SYSA5XXXXV6YJI4W7TBP74KBWBAZH3I6ZGOA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.004400868 (128h16m24.984753515s) [noise=true , address=5.231.73.5:4026 , piece_id=EBTT27VHW4BBJUKZWFVL5TBHI5XUVYN5XQTFRGXYFYOGV46MPVBQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.004728299 (128h16m24.985080947s) [noise=true , address=145.79.227.40:4063 , piece_id=7AZ6XIAGRUM3CSYUPYEQAH6MFVO3K5BSWPHZDT467VTNHFFLSHNQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.004908490 (128h16m24.985261155s) [noise=true , address=145.79.227.11:4746 , piece_id=M47UNXBIAVXN5IO4EW2QLSDJZSAFASHOEYVJEQMQCDAOFROUASIA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.005477487 (128h16m24.985830135s) [noise=true , address=45.91.116.4:4971 , piece_id=JK3LHQNY5YKQDGX4FIAHG4UMUH7X2MQNP2KU5AXCBAZRIXF6AHVA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.005761575 (128h16m24.986114242s) [noise=true , address=45.91.116.13:4185 , piece_id=D7SX42C6L2NRVCNKZ4RQ2DVVSQ3U6KUNYNXU34OIWSF75VYFJIXQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.006056422 (128h16m24.986409068s) [noise=true , address=77.90.31.11:4572 , piece_id=KKVD2YFHQSRPGQFXRXM44TJOPVXNVUKRJNL6HR7HKHY3TT5RCJMQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.006284335 (128h16m24.986636992s) [noise=true , address=145.79.227.26:4683 , piece_id=INAREDRBKMZ4VXMENGVQ3G4KJT3Y7HHLOGTR4R7M6URAHAPFLNQA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.006679725 (128h16m24.987032372s) [noise=true , address=77.90.31.18:4830 , piece_id=K7LIQ6LVEQ4NSTYVSPZ6ZDWEUUSWIACIWW37APSACB3ICTTIV22Q, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.006772481 (128h16m24.987125126s) [noise=true , address=45.91.116.11:4136 , piece_id=HFPECCJUVLBEZBBIRLA5YCAF66SG6EVZZBRSQB3RIQH33NIKDUCQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.007251041 (128h16m24.987603684s) [noise=true , address=145.79.227.29:4959 , piece_id=3LKHZRW4THD3KDKJ32DM7EFM72NOAYU6XPBUUL2JBCPOSFEO2ZRQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.007302090 (128h16m24.987654733s) [noise=true , address=45.91.116.6:4990 , piece_id=KHN5UBCER6NIA6SHEXNBIVPDYWW3ZUIZLTIDICCCWTMMX5JZRPOQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.007752874 (128h16m24.988105579s) [noise=true , address=145.79.227.45:4256 , piece_id=VU27ZQ32SKNLS3CZMYIQPCQEMLF4V63JXFUKCM7TVIVMIHLKYBHQ, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.007931187 (128h16m24.988283836s) [noise=true , address=77.90.31.19:4844 , piece_id=KO4KVUYTCMM7GP4HD5KIWUYA2BHXTK56ZYTBIK45BZM6QYFTMLGA, seg_pos={0 0}, upload=112754]: Uploading piece... Dec 10 23:44:45.564724722 (128h16m25.545077346s) [upload=112754]: Next returned nil segment Dec 10 23:44:45.565804833 (128h16m25.546157472s) [upload=112754]: Waiting for error group managing segments... Dec 10 23:44:45.710962217 (128h16m25.691314864s) [noise=true , address=145.79.227.26:4683 , piece_id=INAREDRBKMZ4VXMENGVQ3G4KJT3Y7HHLOGTR4R7M6URAHAPFLNQA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.711126923 (128h16m25.69147956s) [noise=true , address=145.79.227.11:4746 , piece_id=M47UNXBIAVXN5IO4EW2QLSDJZSAFASHOEYVJEQMQCDAOFROUASIA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.721896521 (128h16m25.702249157s) [noise=true , address=45.91.116.4:4971 , piece_id=JK3LHQNY5YKQDGX4FIAHG4UMUH7X2MQNP2KU5AXCBAZRIXF6AHVA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.730767202 (128h16m25.711119838s) [noise=true , address=45.91.116.6:4990 , piece_id=KHN5UBCER6NIA6SHEXNBIVPDYWW3ZUIZLTIDICCCWTMMX5JZRPOQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.730793588 (128h16m25.711146228s) [noise=true , address=145.79.227.45:4256 , piece_id=VU27ZQ32SKNLS3CZMYIQPCQEMLF4V63JXFUKCM7TVIVMIHLKYBHQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.746147843 (128h16m25.726500479s) [noise=true , address=45.91.116.11:4136 , piece_id=HFPECCJUVLBEZBBIRLA5YCAF66SG6EVZZBRSQB3RIQH33NIKDUCQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.753639014 (128h16m25.733991657s) [noise=true , address=45.91.116.13:4185 , piece_id=D7SX42C6L2NRVCNKZ4RQ2DVVSQ3U6KUNYNXU34OIWSF75VYFJIXQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.754416622 (128h16m25.734769263s) [noise=true , address=145.79.227.40:4063 , piece_id=7AZ6XIAGRUM3CSYUPYEQAH6MFVO3K5BSWPHZDT467VTNHFFLSHNQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.760306170 (128h16m25.740658817s) [noise=true , address=145.79.227.29:4959 , piece_id=3LKHZRW4THD3KDKJ32DM7EFM72NOAYU6XPBUUL2JBCPOSFEO2ZRQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:45.903030317 (128h16m25.883382957s) [noise=true , address=5.231.73.5:4026 , piece_id=EBTT27VHW4BBJUKZWFVL5TBHI5XUVYN5XQTFRGXYFYOGV46MPVBQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:44:47.151249950 (128h16m27.131602592s) [noise=true , address=145.79.227.11:4760 , piece_id=TWP2PR3E2YAXQH5BX7W374VFX26CW26DDL6G4NHNGOANGN7EENOA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: <nil> Dec 10 23:45:44.609511375 (128h17m24.589864017s) [noise=true , address=77.90.31.16:4736 , piece_id=4A4DTFFBSNMAV5PVWEVJQ4JHAOM2KBWSADAHYT3AY2LUX5VEAWEA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:44.609595066 (128h17m24.589947711s) [noise=true , address=77.90.31.19:4844 , piece_id=KO4KVUYTCMM7GP4HD5KIWUYA2BHXTK56ZYTBIK45BZM6QYFTMLGA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: rpc: dial tcp 77.90.31.19:4844: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:44.610095720 (128h17m24.590448365s) [noise=true , address=77.90.31.21:4900 , piece_id=HQCR3WX5ULCE67W6SYSA5XXXXV6YJI4W7TBP74KBWBAZH3I6ZGOA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: rpc: dial tcp 77.90.31.21:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:44.610253504 (128h17m24.590606151s) [noise=true , address=77.90.31.11:4572 , piece_id=KKVD2YFHQSRPGQFXRXM44TJOPVXNVUKRJNL6HR7HKHY3TT5RCJMQ, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:44.610358164 (128h17m24.590710803s) [noise=true , address=77.90.31.12:4619 , piece_id=LXJA4BFZO5XSCQIZIKLQCC4WBZDW4J36MPPNP6UPLOPZPVCG4DTA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:50156->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:44.610660577 (128h17m24.591013219s) [noise=true , address=77.90.31.18:4830 , piece_id=K7LIQ6LVEQ4NSTYVSPZ6ZDWEUUSWIACIWW37APSACB3ICTTIV22Q, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:45:44.610808000 (128h17m24.591160642s) [noise=true , address=77.90.31.15:4731 , piece_id=3CWTL4NSPS2ONXEEZ6AXE5UYOXWBLYEUP255WJQ3HYVULNMEF5AA, seg_pos={0 0}, upload=112754]: Done uploading piece. err: ecclient: upload failed (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, address:77.90.31.15:4731): protocol: write tcp 139.59.144.22:51476->77.90.31.15:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:44.610833695 (128h17m24.591186317s) [seg_pos={0 0}, upload=112754]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:45:44.725231488 (128h17m24.705584146s) [upload=112754]: Done uploading segments... Dec 10 23:45:44.725277965 (128h17m24.705630614s) [upload=112754]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.117909199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................XXX...........................] duration: 1.604196335s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:45:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.604237561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 3.336892486s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:45:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.337008488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................XXX..........................] duration: 1.796192708s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:45:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.796224958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.....X...........] duration: 2.84890075s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:44:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.848928389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.102634254s trace: https://jaeger.dev.storj.io/trace/7e20ea0cd8475ae3 FAIL (2025-12-10 23:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ) [seg_pos={0 0}, upload=112739]: Done begin upload segment. Dec 10 23:43:44.745897465 (128h15m24.726250111s) [noise=true , address=5.231.73.5:4004 , piece_id=LVRYHNA5VXTFHF3DXVI7GVT2EGMMA2JSRTXJID437KXJIFE3NW5Q, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.746050416 (128h15m24.726403058s) [noise=true , address=77.90.31.11:4580 , piece_id=7ZAQIF2475VBCW2RC7TCL26ZFGGPESEO3UT4K4COF4CON6A5HUBQ, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.746761847 (128h15m24.727114503s) [noise=true , address=77.90.31.18:4836 , piece_id=3HJVJWC4Q46HQEAKAT5VMDQTLDG2FZADXKTFUEMR4BZXAAWXX5XA, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.748575901 (128h15m24.728928543s) [noise=true , address=77.90.31.21:4911 , piece_id=653263JQCUZXSHZQ2HDYT7H3FGGS4QMV75AP6MLTU3HGHOFKU6FQ, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.749150387 (128h15m24.729503033s) [noise=true , address=145.79.227.34:4928 , piece_id=D7JHLNCPYYO6HZALYCJ6FJ5QU6JRWBKMEI26UKPZL4JQZNTO6B6Q, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.749770296 (128h15m24.730122939s) [noise=true , address=145.79.227.44:4198 , piece_id=37KFRDRDOAHPTZCZ6DPQYUMOC43YZGSNQ24KCOYUI5H3UIAKOG4Q, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.750220561 (128h15m24.730573189s) [noise=true , address=45.91.116.11:4134 , piece_id=6KJ5LZTDCGP5FVUCWW6AA5DYSAJDVFN7MB4RH6SJWNRQSP4OJOXA, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.750356545 (128h15m24.730709191s) [noise=true , address=45.91.116.9:4273 , piece_id=3PFABKVMQXH2XSZRTXO6RARLBFTBIPRD45LHEKGA6RSY2LRR7B5A, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.750633864 (128h15m24.730986502s) [noise=true , address=145.79.227.38:4986 , piece_id=2LT3PVID26DZZDDZWU447X7ZILF3ZR46GRVJISFG3265JUTNKI3Q, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.750947057 (128h15m24.731299697s) [noise=true , address=64.49.13.139:4477 , piece_id=ZAFUU7EWCXRI3J3NTWSZF4JZQR4N6QG3VST64XXWFKVO3MD5IC7A, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.751033950 (128h15m24.731386577s) [noise=true , address=77.90.31.15:4727 , piece_id=JTXADFQKMGHNFB2OQWRFEZ4RJZAGMKF4BBZQRINF2BHVA6Y67CHQ, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.751595592 (128h15m24.731948271s) [noise=true , address=145.79.227.16:4930 , piece_id=5A6SPFO2D4HWEJ55YF4QTPRK2CSJTFZ6ZMBAZYPBEVAX3PMJLGKQ, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.752266676 (128h15m24.732619326s) [noise=true , address=77.90.31.12:4608 , piece_id=6UPKDETVTDBGEREMCWS7X5UJF6ZSJ4Y6VTHVIA622524L3TRHOOA, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.752882672 (128h15m24.733235325s) [noise=true , address=77.90.31.10:4537 , piece_id=N2TNAC2QERSNLWSHLFFWVMAC25XX62S5P4UOIRW6TMPWMDLZ7CAA, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.753502710 (128h15m24.733855357s) [noise=true , address=145.79.227.25:4649 , piece_id=YCIGRCFWVDGSLAMG2FNKU62BDBHI2DOBNC2KVT2BMMVHW5J52LYQ, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.754098065 (128h15m24.734450716s) [noise=true , address=77.90.31.16:4755 , piece_id=AHSZLWKGSZ2KPHAG365BN4GPMONEWEED26QYWLU5WBGVDZ4LXZ5Q, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.754706398 (128h15m24.735059043s) [noise=true , address=45.91.116.5:4317 , piece_id=I3NPLIQDC6U4BODSFH2ZFZIA2SFD7GVAHZSAARDJBJLNEHHOKQWA, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:44.755297553 (128h15m24.7356502s) [noise=true , address=5.231.73.2:4018 , piece_id=OKBMUSRYGD3QHYVUAN5LXBAMQUWAAR5IWW6Q2CIUG7QZ75FSK23Q, seg_pos={0 0}, upload=112739]: Uploading piece... Dec 10 23:43:45.284579461 (128h15m25.264932098s) [upload=112739]: Next returned nil segment Dec 10 23:43:45.284598091 (128h15m25.264950738s) [upload=112739]: Waiting for error group managing segments... Dec 10 23:43:45.460836228 (128h15m25.441188851s) [noise=true , address=145.79.227.34:4928 , piece_id=D7JHLNCPYYO6HZALYCJ6FJ5QU6JRWBKMEI26UKPZL4JQZNTO6B6Q, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.462114599 (128h15m25.442467243s) [noise=true , address=145.79.227.25:4649 , piece_id=YCIGRCFWVDGSLAMG2FNKU62BDBHI2DOBNC2KVT2BMMVHW5J52LYQ, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.472677227 (128h15m25.453029853s) [noise=true , address=45.91.116.9:4273 , piece_id=3PFABKVMQXH2XSZRTXO6RARLBFTBIPRD45LHEKGA6RSY2LRR7B5A, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.475364083 (128h15m25.455716719s) [noise=true , address=145.79.227.38:4986 , piece_id=2LT3PVID26DZZDDZWU447X7ZILF3ZR46GRVJISFG3265JUTNKI3Q, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.486534231 (128h15m25.466886877s) [noise=true , address=45.91.116.5:4317 , piece_id=I3NPLIQDC6U4BODSFH2ZFZIA2SFD7GVAHZSAARDJBJLNEHHOKQWA, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.487674302 (128h15m25.468026947s) [noise=true , address=45.91.116.11:4134 , piece_id=6KJ5LZTDCGP5FVUCWW6AA5DYSAJDVFN7MB4RH6SJWNRQSP4OJOXA, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.587334821 (128h15m25.567687463s) [noise=true , address=5.231.73.5:4004 , piece_id=LVRYHNA5VXTFHF3DXVI7GVT2EGMMA2JSRTXJID437KXJIFE3NW5Q, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.605507389 (128h15m25.585860028s) [noise=true , address=5.231.73.2:4018 , piece_id=OKBMUSRYGD3QHYVUAN5LXBAMQUWAAR5IWW6Q2CIUG7QZ75FSK23Q, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:45.991161396 (128h15m25.971514035s) [noise=true , address=64.49.13.139:4477 , piece_id=ZAFUU7EWCXRI3J3NTWSZF4JZQR4N6QG3VST64XXWFKVO3MD5IC7A, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:46.641986809 (128h15m26.622339446s) [noise=true , address=145.79.227.16:4930 , piece_id=5A6SPFO2D4HWEJ55YF4QTPRK2CSJTFZ6ZMBAZYPBEVAX3PMJLGKQ, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:43:46.886030822 (128h15m26.866383445s) [noise=true , address=145.79.227.44:4198 , piece_id=37KFRDRDOAHPTZCZ6DPQYUMOC43YZGSNQ24KCOYUI5H3UIAKOG4Q, seg_pos={0 0}, upload=112739]: Done uploading piece. err: <nil> Dec 10 23:44:44.366661653 (128h16m24.347014298s) [noise=true , address=77.90.31.10:4537 , piece_id=N2TNAC2QERSNLWSHLFFWVMAC25XX62S5P4UOIRW6TMPWMDLZ7CAA, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: rpc: dial tcp 77.90.31.10:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:44.366899782 (128h16m24.34725242s) [noise=true , address=77.90.31.11:4580 , piece_id=7ZAQIF2475VBCW2RC7TCL26ZFGGPESEO3UT4K4COF4CON6A5HUBQ, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, address:77.90.31.11:4580): protocol: write tcp 139.59.144.22:43582->77.90.31.11:4580: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:44.367470106 (128h16m24.347822751s) [noise=true , address=77.90.31.18:4836 , piece_id=3HJVJWC4Q46HQEAKAT5VMDQTLDG2FZADXKTFUEMR4BZXAAWXX5XA, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:44.367622987 (128h16m24.347975627s) [noise=true , address=77.90.31.21:4911 , piece_id=653263JQCUZXSHZQ2HDYT7H3FGGS4QMV75AP6MLTU3HGHOFKU6FQ, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: write tcp 139.59.144.22:38528->77.90.31.21:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:44.368117488 (128h16m24.348470124s) [noise=true , address=77.90.31.15:4727 , piece_id=JTXADFQKMGHNFB2OQWRFEZ4RJZAGMKF4BBZQRINF2BHVA6Y67CHQ, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 139.59.144.22:45704->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:44.368400834 (128h16m24.348753495s) [noise=true , address=77.90.31.12:4608 , piece_id=6UPKDETVTDBGEREMCWS7X5UJF6ZSJ4Y6VTHVIA622524L3TRHOOA, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: write tcp 139.59.144.22:51436->77.90.31.12:4608: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:44.368835961 (128h16m24.349188606s) [noise=true , address=77.90.31.16:4755 , piece_id=AHSZLWKGSZ2KPHAG365BN4GPMONEWEED26QYWLU5WBGVDZ4LXZ5Q, seg_pos={0 0}, upload=112739]: Done uploading piece. err: ecclient: upload failed (node:1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, address:77.90.31.16:4755): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:44:44.368888073 (128h16m24.3492407s) [seg_pos={0 0}, upload=112739]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:44:44.467068004 (128h16m24.44742064s) [upload=112739]: Done uploading segments... Dec 10 23:44:44.467114155 (128h16m24.447466794s) [upload=112739]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.102672642s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 2.81073859s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:44:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.810782097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.....X..........] duration: 2.719611382s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:44:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.719635119s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.....X.........] duration: 2.896545702s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:44:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.896625794s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................XXX.........................] duration: 1.669743726s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:43:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.669784028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.109231284s trace: https://jaeger.dev.storj.io/trace/372ddd94fffd7431 FAIL (2025-12-10 23:43:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== gment... Dec 10 23:42:47.790071831 (128h14m27.770424472s) [seg_pos={0 0}, upload=112723]: Done begin upload segment. Dec 10 23:42:47.790103973 (128h14m27.77045662s) [noise=true , address=64.49.13.154:4932 , piece_id=ZVSOFTVO7UTICA5FNERHL7FQTC4674FZEACFPKDY7IMK6L6SBREA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.790200900 (128h14m27.770553537s) [noise=true , address=77.90.31.16:4751 , piece_id=7AZWX22733OSY6FIBZJBM4PCJR5IJXJWCDMDQCP3DD5DXVYUS2DQ, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.791025693 (128h14m27.771378339s) [noise=true , address=77.90.31.11:4567 , piece_id=JQIVSXEGGLAJQKANTJD3MQLMNG6MIXVEZ3766LPV36CML36VMYAA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.791842733 (128h14m27.772195373s) [noise=true , address=145.79.227.31:4823 , piece_id=CYA5VRQTO7HCIQFUC2QFJXEC5NPHM44FYM46FSJW62Z4QYFB64AA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.792047940 (128h14m27.772400582s) [noise=true , address=145.79.227.40:4058 , piece_id=VFDXMOZ65HTVXIQRRXFMEMUIITWHRMQAPQMDPI43PBX7XRR7VOUQ, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.793501798 (128h14m27.773854445s) [noise=true , address=5.231.73.4:4036 , piece_id=N5UVJ53I772O3DZDOBI4GFH7QVBPUN5KFLM27DLTDZRTEKGHHRFA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.793554309 (128h14m27.773906955s) [noise=true , address=77.90.31.10:4535 , piece_id=46KPPBWVQ4Q4GZQXDG6772FUL6BAQEECBXIOXSDMDO6XYYSB63BA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.795307168 (128h14m27.775659814s) [noise=true , address=145.79.227.20:4440 , piece_id=7GGDGXE7YK4HVQ65OOFV3YALPJVY6FACCQLOL645ZCYNWT4CYMGQ, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.795492632 (128h14m27.775845273s) [noise=true , address=145.79.227.16:4926 , piece_id=OL7GED2PP4Q5HOJ5CB55U2QT6MOKDF4DG6HM275J4XLUHNDBLVVA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.796734626 (128h14m27.777087272s) [noise=true , address=77.90.31.11:4587 , piece_id=E3WDVKYDBPHJBJ3JYHPVQ6ZTMJMP6NUOUUK4FCGLF7ZBZKW37BHA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.798112615 (128h14m27.778465252s) [noise=true , address=145.79.227.19:4104 , piece_id=YRG5PZ3KRGOPTNZTLG7RDMS2NC2QAYSO37JWZQXZ36272UBBWE5A, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.798612668 (128h14m27.77896529s) [noise=true , address=145.79.227.44:4198 , piece_id=OBI3MCGX3M3HZYBM3B5K4HADR76NCWEORURI4IN5I2W5PT7ETW4A, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.798672288 (128h14m27.779024935s) [noise=true , address=145.79.227.45:4245 , piece_id=QXWCA2TMTJAVVMVT3YTU6KEE6WZQEJ6UU5ZUBM4V7QN2E64R2KGQ, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.799213394 (128h14m27.77956603s) [noise=true , address=77.90.31.14:4680 , piece_id=J35U5R4C2X6S7CVSJHFCBLWCGPVM6ABWFE6FGQDOARU3C5QTDJ7Q, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.799541978 (128h14m27.779894601s) [noise=true , address=145.79.227.39:4039 , piece_id=VM4LILZ5O4FWNYCV52W7LJ7JFTIZZ57POUYIEL7UFNQVHN46OABA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.799826796 (128h14m27.780179435s) [noise=true , address=145.79.227.30:4779 , piece_id=ZB3OYUURVRHDLTD3YZHBXXTEZXD5RMCJDGYCBDYPJIQTRWEFJRBA, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.800430241 (128h14m27.78078288s) [noise=true , address=77.90.31.19:4862 , piece_id=W5IK2JMFJG5KCJIR4FM4HFP6V64OIZOQ3ZCDEADSDQ5Z47IVDQ3A, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:47.800533286 (128h14m27.780885925s) [noise=true , address=77.90.31.15:4717 , piece_id=NNM3GBMV5J3BMUA53VMAICEMOQTU3JPDBX5LU7PU3LYYEGSGBP2Q, seg_pos={0 0}, upload=112723]: Uploading piece... Dec 10 23:42:48.332719274 (128h14m28.313071911s) [upload=112723]: Next returned nil segment Dec 10 23:42:48.333228754 (128h14m28.313581393s) [upload=112723]: Waiting for error group managing segments... Dec 10 23:42:48.505795507 (128h14m28.486148144s) [noise=true , address=145.79.227.45:4245 , piece_id=QXWCA2TMTJAVVMVT3YTU6KEE6WZQEJ6UU5ZUBM4V7QN2E64R2KGQ, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:48.508683991 (128h14m28.48903663s) [noise=true , address=145.79.227.31:4823 , piece_id=CYA5VRQTO7HCIQFUC2QFJXEC5NPHM44FYM46FSJW62Z4QYFB64AA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:48.508874973 (128h14m28.48922761s) [noise=true , address=145.79.227.39:4039 , piece_id=VM4LILZ5O4FWNYCV52W7LJ7JFTIZZ57POUYIEL7UFNQVHN46OABA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:48.513642270 (128h14m28.49399491s) [noise=true , address=145.79.227.20:4440 , piece_id=7GGDGXE7YK4HVQ65OOFV3YALPJVY6FACCQLOL645ZCYNWT4CYMGQ, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:48.538203399 (128h14m28.518556036s) [noise=true , address=145.79.227.40:4058 , piece_id=VFDXMOZ65HTVXIQRRXFMEMUIITWHRMQAPQMDPI43PBX7XRR7VOUQ, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:48.633849469 (128h14m28.614202105s) [noise=true , address=145.79.227.19:4104 , piece_id=YRG5PZ3KRGOPTNZTLG7RDMS2NC2QAYSO37JWZQXZ36272UBBWE5A, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:48.642637960 (128h14m28.622990597s) [noise=true , address=5.231.73.4:4036 , piece_id=N5UVJ53I772O3DZDOBI4GFH7QVBPUN5KFLM27DLTDZRTEKGHHRFA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:49.039872647 (128h14m29.020225284s) [noise=true , address=64.49.13.154:4932 , piece_id=ZVSOFTVO7UTICA5FNERHL7FQTC4674FZEACFPKDY7IMK6L6SBREA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:49.431106110 (128h14m29.411458738s) [noise=true , address=145.79.227.44:4198 , piece_id=OBI3MCGX3M3HZYBM3B5K4HADR76NCWEORURI4IN5I2W5PT7ETW4A, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:49.513864736 (128h14m29.494217374s) [noise=true , address=145.79.227.16:4926 , piece_id=OL7GED2PP4Q5HOJ5CB55U2QT6MOKDF4DG6HM275J4XLUHNDBLVVA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:42:49.531109782 (128h14m29.511462425s) [noise=true , address=145.79.227.30:4779 , piece_id=ZB3OYUURVRHDLTD3YZHBXXTEZXD5RMCJDGYCBDYPJIQTRWEFJRBA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: <nil> Dec 10 23:43:42.524764856 (128h15m22.505117501s) [noise=true , address=77.90.31.16:4751 , piece_id=7AZWX22733OSY6FIBZJBM4PCJR5IJXJWCDMDQCP3DD5DXVYUS2DQ, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: rpc: dial tcp 77.90.31.16:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:42.524862925 (128h15m22.505215571s) [noise=true , address=77.90.31.15:4717 , piece_id=NNM3GBMV5J3BMUA53VMAICEMOQTU3JPDBX5LU7PU3LYYEGSGBP2Q, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 139.59.144.22:57302->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:42.525178398 (128h15m22.505531043s) [noise=true , address=77.90.31.19:4862 , piece_id=W5IK2JMFJG5KCJIR4FM4HFP6V64OIZOQ3ZCDEADSDQ5Z47IVDQ3A, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 139.59.144.22:54426->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:42.525293513 (128h15m22.505646155s) [noise=true , address=77.90.31.11:4567 , piece_id=JQIVSXEGGLAJQKANTJD3MQLMNG6MIXVEZ3766LPV36CML36VMYAA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:43:42.525357884 (128h15m22.50571053s) [noise=true , address=77.90.31.10:4535 , piece_id=46KPPBWVQ4Q4GZQXDG6772FUL6BAQEECBXIOXSDMDO6XYYSB63BA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 139.59.144.22:39520->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:42.525397723 (128h15m22.505750369s) [noise=true , address=77.90.31.11:4587 , piece_id=E3WDVKYDBPHJBJ3JYHPVQ6ZTMJMP6NUOUUK4FCGLF7ZBZKW37BHA, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: write tcp 139.59.144.22:43888->77.90.31.11:4587: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:42.525451456 (128h15m22.505804102s) [noise=true , address=77.90.31.14:4680 , piece_id=J35U5R4C2X6S7CVSJHFCBLWCGPVM6ABWFE6FGQDOARU3C5QTDJ7Q, seg_pos={0 0}, upload=112723]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:43:42.525495718 (128h15m22.505848355s) [seg_pos={0 0}, upload=112723]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:43:42.630066391 (128h15m22.610419032s) [upload=112723]: Done uploading segments... Dec 10 23:43:42.630112537 (128h15m22.610465176s) [upload=112723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.109248201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................XXX........................] duration: 1.684321554s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.684373563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 2.789132045s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:43:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789205603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................XXX.......................] duration: 1.825481497s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:43:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.825495286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.....X........] duration: 2.787523904s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.787555348s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................XXX......................] duration: 1.609059162s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:42:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.609084957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.113922883s trace: https://jaeger.dev.storj.io/trace/7446f91aae83de32 FAIL (2025-12-10 23:42:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 13m22.563843795s) [upload=112707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:41:42.583508127 (128h13m22.563860763s) [upload=112707]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:41:42.702915258 (128h13m22.683267896s) [seg_pos={0 0}, upload=112707]: Begin upload segment... Dec 10 23:41:42.703015840 (128h13m22.683368487s) [noise=true , address=145.79.227.16:4913 , piece_id=6L27SFDWDGL5GXWSFDQPHPNQTD7WWIVKONXCHBKUTYLWUUW6AIBQ, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.703074544 (128h13m22.68342719s) [seg_pos={0 0}, upload=112707]: Done begin upload segment. Dec 10 23:41:42.703105666 (128h13m22.683458307s) [noise=true , address=45.91.116.13:4176 , piece_id=GTWIBU3TNAYCCZD3KYITKJQ2A4ULSHLV6K7NCH3XYZHZHQNHYYKA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.703704751 (128h13m22.684057388s) [noise=true , address=77.90.31.15:4727 , piece_id=FSQPBLXDWKIM2XRIXRPU4HDJHEFVLNEFDYU2QAGV3MZ4ZQEJSV6Q, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.704738131 (128h13m22.685090773s) [noise=true , address=145.79.227.11:4737 , piece_id=46WEK5XF3NNUMAB3OJNF2XGIMOUF6TJMTEWIPTXBLTEAQ3DMTJ2A, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.705312604 (128h13m22.685665254s) [noise=true , address=45.91.116.5:4330 , piece_id=QDJXBGFM2UZQPQKIOQJRSPCLEPDGLRID45LOHBROFGMEJ2MOEC7A, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.705356294 (128h13m22.685708932s) [noise=true , address=45.91.116.4:4951 , piece_id=VL2RRPC2NFPYLIL34IQAKAE63QQJK35ZE3M5DQUNWCHIGWEL426A, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.705799588 (128h13m22.686152229s) [noise=true , address=77.90.31.11:4574 , piece_id=AJOFGKK4E2OOHTDH2CRUUPE3XPUDJTLSMD53FOAU7HS2UIC4ZJZQ, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.705948952 (128h13m22.68630159s) [noise=true , address=45.91.116.7:4036 , piece_id=IEOD5UKZXV6FNUG7NLIO762NLXFG4MDT2H5YVG3LPG4CQ72QLQ6Q, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.706209494 (128h13m22.686562131s) [noise=true , address=77.90.31.17:4787 , piece_id=MJRPU27OVRBSDCI5JZ4EHE2WD6HTEV5QO7BUYRKVWRYO5YCLWXRA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.706560360 (128h13m22.686913s) [noise=true , address=45.91.116.7:4027 , piece_id=E2ETGG77A3QWKK76HOLUNPRCCK4JSG2VIHBFROPDNQV2GSLHROJQ, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.706896673 (128h13m22.687249313s) [noise=true , address=145.79.227.30:4774 , piece_id=HXFZGP3S2Q5VJLYXBEJYX4ZQ2Z2E5AMH5QLQ4F5AHAFP3IIQ4J4Q, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.707223239 (128h13m22.687575878s) [noise=true , address=145.79.227.28:4748 , piece_id=I6K343DSZRLFOIFIVWOXIAUEGF5P63QVKNTV6PUCYE5OZD52HRCA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.707560572 (128h13m22.687913211s) [noise=true , address=77.90.31.15:4701 , piece_id=DML322I4ECLMCQT4DAVVXVER7AXLASA3O4BZDMGZYTHQQ623RJGA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.707964714 (128h13m22.68831735s) [noise=true , address=145.79.227.25:4640 , piece_id=FVEXJ53BBS57GR56YKXH2Z5DTVI7DPXDS7I4I4HQ4ALX7AJ3B4MQ, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.708398822 (128h13m22.688751467s) [noise=true , address=145.79.227.44:4200 , piece_id=GZMR3LFHY6YYQKD6QVGARMKRVAGGXROCEFTQCPX3OBAUH3BJ7ZUA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.708828282 (128h13m22.689180922s) [noise=true , address=145.79.227.31:4829 , piece_id=LRN3KJLJQKQJQH6AHVYOWZOJSX5W27ORRTDEULK5VLVHQPQKST5A, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.709760028 (128h13m22.690112665s) [noise=true , address=5.231.73.5:4009 , piece_id=UVV6VNYCEU5H6NU7SIMNGMYPCIXFWDNDLLQ6BFAKGQVU5FU24JGA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:42.710375250 (128h13m22.690727879s) [noise=true , address=77.90.31.16:4751 , piece_id=L5VRCEGS4NG7C2OCD73VCNEX76EZOGCJR2SRNTY2O3V2QVH2JNNA, seg_pos={0 0}, upload=112707]: Uploading piece... Dec 10 23:41:43.264972904 (128h13m23.245325541s) [upload=112707]: Next returned nil segment Dec 10 23:41:43.264988157 (128h13m23.245340792s) [upload=112707]: Waiting for error group managing segments... Dec 10 23:41:43.415092453 (128h13m23.39544509s) [noise=true , address=145.79.227.25:4640 , piece_id=FVEXJ53BBS57GR56YKXH2Z5DTVI7DPXDS7I4I4HQ4ALX7AJ3B4MQ, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.415955848 (128h13m23.396308488s) [noise=true , address=145.79.227.28:4748 , piece_id=I6K343DSZRLFOIFIVWOXIAUEGF5P63QVKNTV6PUCYE5OZD52HRCA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.426513926 (128h13m23.406866564s) [noise=true , address=45.91.116.5:4330 , piece_id=QDJXBGFM2UZQPQKIOQJRSPCLEPDGLRID45LOHBROFGMEJ2MOEC7A, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.427654918 (128h13m23.408007564s) [noise=true , address=145.79.227.11:4737 , piece_id=46WEK5XF3NNUMAB3OJNF2XGIMOUF6TJMTEWIPTXBLTEAQ3DMTJ2A, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.428393742 (128h13m23.408746378s) [noise=true , address=145.79.227.31:4829 , piece_id=LRN3KJLJQKQJQH6AHVYOWZOJSX5W27ORRTDEULK5VLVHQPQKST5A, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.432457476 (128h13m23.412810114s) [noise=true , address=45.91.116.4:4951 , piece_id=VL2RRPC2NFPYLIL34IQAKAE63QQJK35ZE3M5DQUNWCHIGWEL426A, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.434187574 (128h13m23.414540212s) [noise=true , address=45.91.116.7:4036 , piece_id=IEOD5UKZXV6FNUG7NLIO762NLXFG4MDT2H5YVG3LPG4CQ72QLQ6Q, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.441823687 (128h13m23.422176313s) [noise=true , address=45.91.116.7:4027 , piece_id=E2ETGG77A3QWKK76HOLUNPRCCK4JSG2VIHBFROPDNQV2GSLHROJQ, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.453809523 (128h13m23.43416216s) [noise=true , address=45.91.116.13:4176 , piece_id=GTWIBU3TNAYCCZD3KYITKJQ2A4ULSHLV6K7NCH3XYZHZHQNHYYKA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:43.661348035 (128h13m23.641700671s) [noise=true , address=5.231.73.5:4009 , piece_id=UVV6VNYCEU5H6NU7SIMNGMYPCIXFWDNDLLQ6BFAKGQVU5FU24JGA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:44.339150589 (128h13m24.319503228s) [noise=true , address=145.79.227.30:4774 , piece_id=HXFZGP3S2Q5VJLYXBEJYX4ZQ2Z2E5AMH5QLQ4F5AHAFP3IIQ4J4Q, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:44.413340680 (128h13m24.393693318s) [noise=true , address=145.79.227.44:4200 , piece_id=GZMR3LFHY6YYQKD6QVGARMKRVAGGXROCEFTQCPX3OBAUH3BJ7ZUA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:41:44.653637689 (128h13m24.633990325s) [noise=true , address=145.79.227.16:4913 , piece_id=6L27SFDWDGL5GXWSFDQPHPNQTD7WWIVKONXCHBKUTYLWUUW6AIBQ, seg_pos={0 0}, upload=112707]: Done uploading piece. err: <nil> Dec 10 23:42:42.266033855 (128h14m22.2463865s) [noise=true , address=77.90.31.15:4701 , piece_id=DML322I4ECLMCQT4DAVVXVER7AXLASA3O4BZDMGZYTHQQ623RJGA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 139.59.144.22:54162->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:42.266138345 (128h14m22.246490987s) [noise=true , address=77.90.31.11:4574 , piece_id=AJOFGKK4E2OOHTDH2CRUUPE3XPUDJTLSMD53FOAU7HS2UIC4ZJZQ, seg_pos={0 0}, upload=112707]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 139.59.144.22:47888->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:42.266204850 (128h14m22.24655749s) [noise=true , address=77.90.31.15:4727 , piece_id=FSQPBLXDWKIM2XRIXRPU4HDJHEFVLNEFDYU2QAGV3MZ4ZQEJSV6Q, seg_pos={0 0}, upload=112707]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 139.59.144.22:39452->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:42.266323538 (128h14m22.246676178s) [noise=true , address=77.90.31.17:4787 , piece_id=MJRPU27OVRBSDCI5JZ4EHE2WD6HTEV5QO7BUYRKVWRYO5YCLWXRA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:43650->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:42.266516411 (128h14m22.246869052s) [noise=true , address=77.90.31.16:4751 , piece_id=L5VRCEGS4NG7C2OCD73VCNEX76EZOGCJR2SRNTY2O3V2QVH2JNNA, seg_pos={0 0}, upload=112707]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: write tcp 139.59.144.22:56790->77.90.31.16:4751: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:42.266547265 (128h14m22.246899891s) [seg_pos={0 0}, upload=112707]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:42:42.377654389 (128h14m22.358007031s) [upload=112707]: Done uploading segments... Dec 10 23:42:42.377714383 (128h14m22.358067027s) [upload=112707]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.113936344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 2.820959433s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:42:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.821054805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XXX.....................] duration: 1.698274205s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.698376708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.....X.......] duration: 2.698574417s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:42:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.698606349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.111078551s trace: https://jaeger.dev.storj.io/trace/6243bc2dbdb7b9c7 FAIL (2025-12-10 23:41:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ec 10 23:40:42.428527495 (128h12m22.408880142s) [seg_pos={0 0}, upload=112691]: Begin upload segment... Dec 10 23:40:42.428611819 (128h12m22.408964459s) [seg_pos={0 0}, upload=112691]: Done begin upload segment. Dec 10 23:40:42.428683110 (128h12m22.409035763s) [noise=true , address=77.90.31.12:4614 , piece_id=B3KFRBPZZLOAFGTX7ZD7ELHQYA2ZO2HQ6CFF4KKHF64PQWS5UIAQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.429811746 (128h12m22.410164393s) [noise=true , address=145.79.227.25:4636 , piece_id=WDHKGUOFN6W7ZN7ZPBDWVCBFJFUGRN5N6W5XQXKZIPQBD64T3WVQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.430170298 (128h12m22.410522939s) [noise=true , address=64.49.13.155:4913 , piece_id=JJFPOEKQS25RETAHQ3K5XUJWICX4SJCQ643EGG6DE2RQBZAD5DRQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.430465817 (128h12m22.410818458s) [noise=true , address=145.79.227.21:4498 , piece_id=LPTB66TWMVEL3R356CN32P4PRK3YK2TC5MFCLGWVTQVSFM2IHPQA, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.430735988 (128h12m22.411088629s) [noise=true , address=45.91.116.9:4282 , piece_id=WRGEKFKHMLGV66L5YFVDZSL75N7543XBUS3WTNQIEUM6ZHMSSQNQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.431055900 (128h12m22.411408541s) [noise=true , address=77.90.31.13:4648 , piece_id=I4A5GBO5RECEXWTDCLQN7LDCS6FIHW5OUQACXKY7S5QVZPJBOVXQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.431340845 (128h12m22.411693482s) [noise=true , address=77.90.31.18:4833 , piece_id=CU45TDWDIA7A3BUNQ3ZFE23PABLABGECWUBD72Y6H56BRQV6PANA, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.431633139 (128h12m22.41198578s) [noise=true , address=145.79.227.16:4929 , piece_id=HE42MHRV2TSW6C66NG5462F4CTSRTC427LLQS23ASYRJLXFYUB7Q, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.432013084 (128h12m22.412365724s) [noise=true , address=45.91.116.8:4053 , piece_id=YTCBWYFKYK4DTYQLCVXYUOQLNIMQBHRWIOIMMW26EEYUB64GJOBQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.433984475 (128h12m22.414337112s) [noise=true , address=77.90.31.21:4886 , piece_id=GPIKH5V6KYF3XJMINLTD4VINPADP2FQLEWRELE27KN7HIM35T7RQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.434052276 (128h12m22.414404928s) [noise=true , address=45.91.116.9:4287 , piece_id=QQV22RDML6H4UZHNTYMJPNDLABESR2CE2WPHFTXFXIOHKOI6UHPQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.434676991 (128h12m22.415029629s) [noise=true , address=77.90.31.15:4723 , piece_id=ATOLBZL3WYNNAELPKK5T7GSXRSAO2KKLETGHJ2SPX6BNVQCZJIQA, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.434693483 (128h12m22.415046131s) [noise=true , address=77.90.31.14:4693 , piece_id=KL72MUJ64V3UB3PBIQTTDWICKCOLSCFWZRMHAZKQK4WYSXRI3DRQ, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.435227672 (128h12m22.41558032s) [noise=true , address=45.91.116.4:4956 , piece_id=V25OJNCR4MLYI2LOIWDA4VR5GJKDVSTHZD32LJOMSPUBBR7NJR7Q, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.435297596 (128h12m22.415650237s) [noise=true , address=145.79.227.24:4585 , piece_id=XEZ4WCUFHAXGKFWMLHKXYOSVPAGZDCSCXVZHNFB3OHHJ36GJU64A, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.435815242 (128h12m22.416167892s) [noise=true , address=77.90.31.17:4796 , piece_id=SBGFI2MSEW6SAYYYF5X257TD4BF7SFTIFRQ2KXLVLPJLOSOXWN7A, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.435922903 (128h12m22.416275543s) [noise=true , address=45.91.116.10:4110 , piece_id=ESQ6P5Y6ODQADSPQZUSD6BMPLI47ADDNEUF4PCQEWR4XLGOW5X4A, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.436544771 (128h12m22.416897412s) [noise=true , address=45.91.116.12:4253 , piece_id=72AP2JQ77YBWURFA7HEUQV3NEV6PTQO6T4ZFTN4C7N2DHVQC6A6A, seg_pos={0 0}, upload=112691]: Uploading piece... Dec 10 23:40:42.967918771 (128h12m22.94827141s) [upload=112691]: Next returned nil segment Dec 10 23:40:42.967945941 (128h12m22.948298589s) [upload=112691]: Waiting for error group managing segments... Dec 10 23:40:43.138652024 (128h12m23.119004667s) [noise=true , address=145.79.227.21:4498 , piece_id=LPTB66TWMVEL3R356CN32P4PRK3YK2TC5MFCLGWVTQVSFM2IHPQA, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.144591736 (128h12m23.124944374s) [noise=true , address=45.91.116.10:4110 , piece_id=ESQ6P5Y6ODQADSPQZUSD6BMPLI47ADDNEUF4PCQEWR4XLGOW5X4A, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.144802608 (128h12m23.125155237s) [noise=true , address=145.79.227.24:4585 , piece_id=XEZ4WCUFHAXGKFWMLHKXYOSVPAGZDCSCXVZHNFB3OHHJ36GJU64A, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.145453622 (128h12m23.125806262s) [noise=true , address=45.91.116.12:4253 , piece_id=72AP2JQ77YBWURFA7HEUQV3NEV6PTQO6T4ZFTN4C7N2DHVQC6A6A, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.150174871 (128h12m23.130527524s) [noise=true , address=45.91.116.9:4282 , piece_id=WRGEKFKHMLGV66L5YFVDZSL75N7543XBUS3WTNQIEUM6ZHMSSQNQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.161523778 (128h12m23.141876404s) [noise=true , address=45.91.116.9:4287 , piece_id=QQV22RDML6H4UZHNTYMJPNDLABESR2CE2WPHFTXFXIOHKOI6UHPQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.162791690 (128h12m23.143144335s) [noise=true , address=45.91.116.4:4956 , piece_id=V25OJNCR4MLYI2LOIWDA4VR5GJKDVSTHZD32LJOMSPUBBR7NJR7Q, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.163592442 (128h12m23.143945083s) [noise=true , address=45.91.116.8:4053 , piece_id=YTCBWYFKYK4DTYQLCVXYUOQLNIMQBHRWIOIMMW26EEYUB64GJOBQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.164555449 (128h12m23.144908091s) [noise=true , address=145.79.227.25:4636 , piece_id=WDHKGUOFN6W7ZN7ZPBDWVCBFJFUGRN5N6W5XQXKZIPQBD64T3WVQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:43.823807960 (128h12m23.804160606s) [noise=true , address=64.49.13.155:4913 , piece_id=JJFPOEKQS25RETAHQ3K5XUJWICX4SJCQ643EGG6DE2RQBZAD5DRQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:40:44.198174269 (128h12m24.178526932s) [noise=true , address=145.79.227.16:4929 , piece_id=HE42MHRV2TSW6C66NG5462F4CTSRTC427LLQS23ASYRJLXFYUB7Q, seg_pos={0 0}, upload=112691]: Done uploading piece. err: <nil> Dec 10 23:41:42.023878776 (128h13m22.004231423s) [noise=true , address=77.90.31.12:4614 , piece_id=B3KFRBPZZLOAFGTX7ZD7ELHQYA2ZO2HQ6CFF4KKHF64PQWS5UIAQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:42.023901735 (128h13m22.004254382s) [noise=true , address=77.90.31.15:4723 , piece_id=ATOLBZL3WYNNAELPKK5T7GSXRSAO2KKLETGHJ2SPX6BNVQCZJIQA, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: rpc: dial tcp 77.90.31.15:4723: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:42.024228785 (128h13m22.004581431s) [noise=true , address=77.90.31.18:4833 , piece_id=CU45TDWDIA7A3BUNQ3ZFE23PABLABGECWUBD72Y6H56BRQV6PANA, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, address:77.90.31.18:4833): protocol: rpc: dial tcp 77.90.31.18:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:42.024524913 (128h13m22.004877558s) [noise=true , address=77.90.31.14:4693 , piece_id=KL72MUJ64V3UB3PBIQTTDWICKCOLSCFWZRMHAZKQK4WYSXRI3DRQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:41:42.024811641 (128h13m22.005164282s) [noise=true , address=77.90.31.17:4796 , piece_id=SBGFI2MSEW6SAYYYF5X257TD4BF7SFTIFRQ2KXLVLPJLOSOXWN7A, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:42.024840719 (128h13m22.005193362s) [noise=true , address=77.90.31.13:4648 , piece_id=I4A5GBO5RECEXWTDCLQN7LDCS6FIHW5OUQACXKY7S5QVZPJBOVXQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: rpc: dial tcp 77.90.31.13:4648: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:42.025181832 (128h13m22.005534474s) [noise=true , address=77.90.31.21:4886 , piece_id=GPIKH5V6KYF3XJMINLTD4VINPADP2FQLEWRELE27KN7HIM35T7RQ, seg_pos={0 0}, upload=112691]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: rpc: dial tcp 77.90.31.21:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:42.025219398 (128h13m22.005572035s) [seg_pos={0 0}, upload=112691]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:41:42.133497448 (128h13m22.113850085s) [upload=112691]: Done uploading segments... Dec 10 23:41:42.133537361 (128h13m22.113890003s) [upload=112691]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.111095831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 2.973170685s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:41:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.973199398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XXX....................] duration: 1.670202671s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:41:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.670222162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.....X......] duration: 2.705547227s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:41:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.705569493s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.....X.....] duration: 2.821556764s trace: https://jaeger.dev.storj.io/trace/31a3a014d0a692b6 FAIL (2025-12-10 23:41:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.821586133s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 2.743250677s trace: https://jaeger.dev.storj.io/trace/1d94eac5699bd6c5 FAIL (2025-12-10 23:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.743285051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................XXX...................] duration: 1.726550978s trace: https://jaeger.dev.storj.io/trace/4d18ebf192707efc FAIL (2025-12-10 23:40:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.726590175s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)