Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 2.786519999s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:27:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786590639s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.334756148s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:27:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.334787767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.316911343s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:26:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.316941325s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 2.711642868s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:26:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712328543s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.263038389s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:26:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.263071454s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 2.555800999s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:25:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.555829848s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 2.667684904s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:25:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.667746711s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 2.79527921s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:25:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.795307149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.413846459s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:24:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.413876184s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 2.577103789s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:24:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.577138939s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.312134912s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:24:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.312152859s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 2.73312338s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:23:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.733151777s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.356329627s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:23:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.356361345s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 2.514961427s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:23:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.514995195s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 2.65978374s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:22:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.659936068s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.424158376s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:22:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.424186113s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 2.623037174s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:22:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.623073428s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.352723892s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:21:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.352755192s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 2.784997763s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:21:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.785029969s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.4071559s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:21:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.40718775s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 2.745419885s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:20:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.745456922s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.340607642s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:20:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.340743502s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 2.525717698s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:19:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.525796543s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 2.557593142s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:19:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.557621388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.5862755s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:19:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.586320466s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.712735474s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:18:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712765281s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.41652986s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:18:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.416560633s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 2.647382333s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:17:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.647407936s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.325242755s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:17:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.325267193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.334893175s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:17:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.334922386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.343327692s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:16:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.343349467s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 2.688752397s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.688786625s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.446808068s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:16:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.44683694s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 2.538765063s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:15:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.538797366s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.296329408s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:15:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.296354182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.225310758s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:14:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.225338737s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 2.467279788s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.467307393s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 2.463383504s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:14:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.463409969s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.515139389s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:13:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.515173011s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 2.812749494s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:13:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.81280518s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 2.502940708s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:13:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.502978408s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.453413862s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:12:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.453441328s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 2.925981083s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:12:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.925998688s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 2.61802907s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:11:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.61805488s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.374627902s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:11:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.374664001s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.60419274s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:11:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.60511868s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.339345984s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:10:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.33939488s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 3.398844663s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:10:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398870721s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 18.813030514s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:10:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.813063301s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.155585622s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 19.15561562s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)