Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.752135872s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:45:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.752161083s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.375141711s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:45:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.375169764s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.604920726s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:44:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.604948709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.334127256s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.334181303s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.628025384s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:43:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.628052368s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.315753929s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:43:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.31578277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.361499352s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:43:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.361527069s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 2.691016396s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:42:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.691067099s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.2930533s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.293085671s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 2.743307705s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:41:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.743340625s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.385097007s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:41:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.385128195s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 2.786691075s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:41:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786852538s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 2.752674621s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:40:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.752762924s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.356122142s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:40:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.356156991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.460654651s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.460690797s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 2.721234425s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:39:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.721265353s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.304693577s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:39:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.304726172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.274054874s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:39:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.274080381s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 2.925580979s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.925606303s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 2.830275581s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:38:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.830301361s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.377141807s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:38:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.377171495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.410951256s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.410984353s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 2.669253485s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:38:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.669275822s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 2.64392048s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:37:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.643962381s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.377001023s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:37:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.377020612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.334661117s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:36:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.334710605s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 2.741034164s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:36:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.741064701s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.410338695s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:35:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.410371494s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 2.647134977s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:35:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.647238859s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 2.694168964s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:35:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.694198572s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.309217638s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:34:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.309255643s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 2.67641722s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:34:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.676444734s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.529607677s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:34:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.529694217s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 2.907220378s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:33:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.907258994s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.432980722s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:33:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.433002883s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 2.635567431s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:33:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.63559575s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.293078006s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.293105757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 2.768670667s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:31:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.768698307s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.427536064s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:31:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.427562943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.474138163s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:31:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.474172215s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 2.678764273s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:31:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.678800846s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 2.806663824s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:30:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.806688914s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.224566437s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:29:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.224591109s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 2.554400058s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:29:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.554442044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.404267712s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.40429663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.354341041s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.354372122s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 2.555271499s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:28:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.55529838s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 2.788450217s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:28:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.788500173s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.425506556s trace: https://jaeger.dev.storj.io/trace/73a29de9c809b554 FAIL (2025-11-05 07:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 53.391572844 (139h27m56.093165284s) [upload=122653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 07:08:53.511496551 (139h27m56.213088992s) [seg_pos={0 0}, upload=122653]: Begin upload segment... Nov 5 07:08:53.511554279 (139h27m56.21314672s) [seg_pos={0 0}, upload=122653]: Done begin upload segment. Nov 5 07:08:53.511581616 (139h27m56.213174066s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513588552 (139h27m56.215181007s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.513752850 (139h27m56.215345301s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.514354806 (139h27m56.215947257s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515244711 (139h27m56.216837159s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.515784410 (139h27m56.217376865s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516021191 (139h27m56.217613644s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516273977 (139h27m56.217866426s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516440796 (139h27m56.218033244s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516682821 (139h27m56.218275269s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.516799834 (139h27m56.218392285s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517334963 (139h27m56.218927412s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517377298 (139h27m56.21896976s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.517781951 (139h27m56.2193744s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518271127 (139h27m56.219863578s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:53.518768591 (139h27m56.220361047s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Uploading piece... Nov 5 07:08:54.140079499 (139h27m56.841671947s) [upload=122653]: Next returned nil segment Nov 5 07:08:54.140092295 (139h27m56.84168473s) [upload=122653]: Waiting for error group managing segments... Nov 5 07:08:54.275546767 (139h27m56.977139218s) [noise=true , address=77.90.31.18:4807 , piece_id=2OGMRA7RUWBJOKEN3MNZLTW6VIXPOLHRR3O5H3CVNT4AUEKWBXBQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285318656 (139h27m56.98691112s) [noise=true , address=77.90.31.13:4628 , piece_id=L3WCZYDMEM2W2552M7LG2A5PBP4HU64JNQMCJAENRC46BUAOG3EQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.285563903 (139h27m56.987156362s) [noise=true , address=77.90.31.18:4805 , piece_id=VUQEVDQ3UPZFATAGOQ4QSMTCSWFPMBEK4MRZLF4VOKPLFPYIMMBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.307243468 (139h27m57.008835919s) [noise=true , address=77.90.31.21:4879 , piece_id=DTTPJA34K7VYRBAGQ6LYIPKNR4ZUFWNU3XICCXFSIS5VCOCFRGRA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.311638805 (139h27m57.013231286s) [noise=true , address=77.90.31.14:4663 , piece_id=TBLQUWZALDARTQIBY3TYWHK24QQYSRM47NCDFDYI3ROL2P2WFIIA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:08:54.403790160 (139h27m57.105382594s) [noise=true , address=77.90.31.12:4590 , piece_id=DEMANEV6ZWO62EVVT6FXPJOTNYQLPJIC6ABQCTJCF2MPI2PF2MCA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: <nil> Nov 5 07:09:23.097144333 (139h28m25.7987368s) [noise=true , address=145.79.227.21:4475 , piece_id=42BQHCUFBK2SODY3PIVRYD4N3CQRXDZ6CHXZP3OM2BGRPYBDTI7Q, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: rpc: dial tcp 145.79.227.21:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097392075 (139h28m25.798984538s) [noise=true , address=145.79.227.27:4690 , piece_id=3ZJPN6QKPUH4RIVRTPOJOV6Y4FD4YKYQHZJGAV3CUNWOUTLUY3ZA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: rpc: dial tcp 145.79.227.27:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097504503 (139h28m25.799096966s) [noise=true , address=145.79.227.24:4582 , piece_id=SM2SI6H445LDLWKN4433SZ2JWNKW7VWCUZD6QI5DC72FFKXIZKGA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: rpc: dial tcp 145.79.227.24:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097732944 (139h28m25.799325406s) [noise=true , address=145.79.227.32:4834 , piece_id=NIF4YDGRWHNM734KXYVCGWWHNQPVPZIWZD5UGJKEWGMKWUYN634A, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: rpc: dial tcp 145.79.227.32:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.097846391 (139h28m25.799438853s) [noise=true , address=145.79.227.13:4802 , piece_id=PQEQCX24FLF7WNZL6NVRSBH2E4IESGGMEESF77XLCL2DZXTML6DA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098108991 (139h28m25.799701452s) [noise=true , address=145.79.227.10:4401 , piece_id=E3TWGH2FG7QBTIQMEZ3TVTHRQCRUPVJGF65RSPUPF3M6NKRNZHHA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098205549 (139h28m25.799798009s) [noise=true , address=145.79.227.15:4876 , piece_id=P65E3AEIWXVODXYXRLYI6CCUCL37W4LVPVZFE7RYG3L6KJ7SMJVQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098453405 (139h28m25.80004587s) [noise=true , address=145.79.227.18:4005 , piece_id=5AKQXEJY72OR6NAVFSE5S6EX2TQUURE6R4JCYFWOZNNULR5SS6FQ, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098482863 (139h28m25.800075324s) [noise=true , address=145.79.227.31:4797 , piece_id=4IXOFVR7EBRFFWAQKCLNWGE7J5ZARFXBJQ67A3H7FUEP2IQYKNBA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: rpc: dial tcp 145.79.227.31:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098774913 (139h28m25.800367375s) [noise=true , address=145.79.227.14:4840 , piece_id=JVAE4ZXJ6MX4KFVAXDYKSXDYJ5EMNCGMBXRK3WVIPD6NGWB2GQDA, seg_pos={0 0}, upload=122653]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 07:09:23.098823983 (139h28m25.800416425s) [seg_pos={0 0}, upload=122653]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 5 07:09:23.210562291 (139h28m25.912154746s) [upload=122653]: Done uploading segments... Nov 5 07:09:23.210617929 (139h28m25.912210379s) [upload=122653]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 2.42560623s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 2.718093994s trace: https://jaeger.dev.storj.io/trace/49cd51d016f1caf5 FAIL (2025-11-05 07:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.718124803s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)