Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 2.614136814s FAIL (2025-06-26 06:21:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.614168316s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 2.295495407s FAIL (2025-06-26 06:20:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.295530577s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 2.840495006s FAIL (2025-06-26 06:20:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.841144438s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 2.748709462s FAIL (2025-06-26 06:20:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.74879285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 2.662492415s FAIL (2025-06-26 06:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.662591089s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.342117644s FAIL (2025-06-26 06:19:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.342151528s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 2.655148231s FAIL (2025-06-26 06:19:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.655174163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 2.353625688s FAIL (2025-06-26 06:18:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.353657608s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 2.675210219s FAIL (2025-06-26 06:18:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.675236968s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 2.497349881s FAIL (2025-06-26 06:18:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.49738237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 3.062073367s FAIL (2025-06-26 06:18:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.062108127s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 2.297740033s FAIL (2025-06-26 06:17:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.297782272s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.813019952s FAIL (2025-06-26 06:17:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.813043523s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 2.667972238s FAIL (2025-06-26 06:16:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.66799119s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 2.402977991s FAIL (2025-06-26 06:16:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.403010423s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 2.581114532s FAIL (2025-06-26 06:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.581143247s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 2.376340578s FAIL (2025-06-26 06:15:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.376373584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 2.280456171s FAIL (2025-06-26 06:14:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.28048727s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 2.658171953s FAIL (2025-06-26 06:14:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.658207944s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 2.256751884s FAIL (2025-06-26 06:14:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.256847121s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 2.813614973s FAIL (2025-06-26 06:14:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.813637441s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 2.64189811s FAIL (2025-06-26 06:13:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.641948726s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 2.377946814s FAIL (2025-06-26 06:13:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.37797649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 2.283729435s FAIL (2025-06-26 06:12:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.283906525s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 2.717626831s FAIL (2025-06-26 06:12:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.717674307s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 2.558440854s FAIL (2025-06-26 06:11:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.558462909s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.398133102s FAIL (2025-06-26 06:11:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.398159316s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 2.632882985s FAIL (2025-06-26 06:11:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.632911629s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 2.361772692s FAIL (2025-06-26 06:10:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.361803417s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 2.772866488s FAIL (2025-06-26 06:10:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.772906381s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.947141492s FAIL (2025-06-26 06:09:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.947165929s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 3.404134396s FAIL (2025-06-26 06:09:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 3.404167801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.108130349s FAIL (2025-06-26 06:09:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ed. err: failed to upload enough pieces (needed at least 13 but got 11) Jun 26 06:08:46.149410115 (68h57m46.032823751s) [upload=55164]: Starting upload Jun 26 06:08:46.149511905 (68h57m46.032925531s) [upload=55164]: Done starting upload Jun 26 06:08:46.149795315 (68h57m46.033208934s) [upload=55164]: Uploading segments... Jun 26 06:08:46.149856673 (68h57m46.033270291s) [upload=55164]: Got next segment. Inline: false Jun 26 06:08:46.149949372 (68h57m46.033362999s) [upload=55164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:46.149955991 (68h57m46.033369622s) [upload=55164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:46.256737978 (68h57m46.140151591s) [seg_pos={0 0}, upload=55164]: Begin upload segment... Jun 26 06:08:46.256791847 (68h57m46.140205461s) [seg_pos={0 0}, upload=55164]: Done begin upload segment. Jun 26 06:08:46.256817169 (68h57m46.140230797s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.256845089 (68h57m46.140258718s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257153939 (68h57m46.140567569s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257402464 (68h57m46.140816091s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.257702039 (68h57m46.141115666s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.258771691 (68h57m46.14218533s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.259903667 (68h57m46.143317298s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260178371 (68h57m46.143591997s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260466182 (68h57m46.143879836s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.260753254 (68h57m46.144166867s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261021742 (68h57m46.144435369s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261275360 (68h57m46.144688989s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.261526496 (68h57m46.144940122s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262053155 (68h57m46.14546678s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262502914 (68h57m46.145916543s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.262903705 (68h57m46.146317334s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Uploading piece... Jun 26 06:08:46.864576292 (68h57m46.74798991s) [upload=55164]: Next returned nil segment Jun 26 06:08:46.864595177 (68h57m46.748008793s) [upload=55164]: Waiting for error group managing segments... Jun 26 06:08:47.003788840 (68h57m46.88720247s) [noise=true , address=104.36.82.16:4294 , piece_id=UJZR7PVKMQHLVCG7LYJP6X4XO2UIV2V36WLGQ6G5VZNUDKN5GRTQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.005691175 (68h57m46.889104787s) [noise=true , address=104.36.82.13:4187 , piece_id=7TTSKMLMBKTCMVCDZF4IF5LUREE625KNTNT7B2DT7L2UIHPBU3TQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.007200494 (68h57m46.890614107s) [noise=true , address=104.36.82.11:4115 , piece_id=PMIRDCZITRUNULQCVTHC5EMJRJNYAVCUAXBTWVGMIHPPSGVLTPLA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008311260 (68h57m46.891724874s) [noise=true , address=104.36.82.13:4185 , piece_id=Y4PKYNZGTJE7AARTXY6VDXHHWEEMV2JZEQZWBPHG2PM26EIUD7CQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.008467994 (68h57m46.891881608s) [noise=true , address=104.36.82.2:4729 , piece_id=RQ4OP57NLIB2ANTMIHJXVGOYGC6MW53D3CTJJH6ES7MT5KPCY4SA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.009777908 (68h57m46.893191536s) [noise=true , address=104.36.82.10:4078 , piece_id=4LZBTFKJSUJMZLQARME6BXVB3H5WESFZNW25YJ2R6GHZT2OXL24A, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.043010535 (68h57m46.92642415s) [noise=true , address=104.36.82.12:4151 , piece_id=WOKRYNB4PN7ECNIBJE3NFGHFAGWI76RFRWZZVI2INAEK6WDKIGDA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.047598840 (68h57m46.931012476s) [noise=true , address=104.36.82.7:4912 , piece_id=BU4CEFCBKQT7AWQZUAZPOHSVMUXNNDKQTSC7C56XRVRWNNP6DO7Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:08:47.193461295 (68h57m47.076874929s) [noise=true , address=104.36.82.5:4839 , piece_id=UWT4YOSX56AFHVPNI2QCMZBWQBE5DR4744U4F626VBGLYWE5MTWA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: <nil> Jun 26 06:09:15.876852535 (68h58m15.760266169s) [noise=true , address=45.91.116.7:4009 , piece_id=HBHKYHIZEVAZOMQZSMXAOHVRS2AZTUKKCBOZSE6BKICLBD3YX4MQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.876933547 (68h58m15.760347166s) [noise=true , address=45.91.116.4:4939 , piece_id=6772RQD2SQSIXK5Y5MIC3TI3YLPF7YHK4HU2C46S7ZGTDFW2A2QQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:42128->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877005614 (68h58m15.760419242s) [noise=true , address=77.90.31.15:4699 , piece_id=QGFTD3CAGNOSOERNCSQJY2HSQ32QPJYLCDM6EZ3J46427YMAPOJA, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:34760->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877091107 (68h58m15.760504723s) [noise=true , address=45.91.116.5:5298 , piece_id=42RTBIB7AC4ESAFENJA42VCI53PZ6IYXFF7YWXCCCEOHPTYKHQ6Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 139.59.144.22:38610->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877252123 (68h58m15.760665755s) [noise=true , address=45.91.116.10:4083 , piece_id=QAS5LP63LNLVPTV2HXP2ISP5GOF6434XIWBWAJDD45SZYXY6RE3Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 139.59.144.22:39986->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877335452 (68h58m15.760749085s) [noise=true , address=77.90.31.17:4772 , piece_id=CVT4UOXEQUFIJFISCYFLZFMN5TZTULOINTVGRWZP3KK5WFCJRYCQ, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 139.59.144.22:57926->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877412512 (68h58m15.760826142s) [noise=true , address=77.90.31.12:4590 , piece_id=PU3VK4OCFSSUP7SQOD2EF2FN2F24OZZM5IJ45XFNTKDI5ZDLJT2Q, seg_pos={0 0}, upload=55164]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:39360->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:09:15.877469294 (68h58m15.760882924s) [seg_pos={0 0}, upload=55164]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jun 26 06:09:15.983403909 (68h58m15.866817535s) [upload=55164]: Done uploading segments... Jun 26 06:09:15.983433081 (68h58m15.866846707s) [upload=55164]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.108160855s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 2.789114339s FAIL (2025-06-26 06:09:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789142997s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 2.492171452s FAIL (2025-06-26 06:08:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.492185534s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001079154s FAIL (2025-06-26 06:08:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001095295s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.108314507s FAIL (2025-06-26 06:08:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 6XXE5P66AXA, seg_pos={0 0}, upload=55135]: Done uploading piece. err: <nil> Jun 26 06:06:33.776727906 (68h55m33.660141544s) [seg_pos={0 0}, upload=55135]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jun 26 06:06:33.776846722 (68h55m33.660260361s) [upload=55135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 26 06:06:33.776852700 (68h55m33.66026634s) [upload=55135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 26 06:06:33.930618503 (68h55m33.814032141s) [upload=55135]: Done uploading segments... Jun 26 06:06:33.930660393 (68h55m33.814074026s) [upload=55135]: Upload finished. err: <nil> Jun 26 06:07:34.198715474 (68h56m34.082129092s) [upload=55149]: Starting upload Jun 26 06:07:34.198904880 (68h56m34.082318519s) [upload=55149]: Done starting upload Jun 26 06:07:34.199222121 (68h56m34.082635738s) [upload=55149]: Uploading segments... Jun 26 06:07:34.199291164 (68h56m34.082704809s) [upload=55149]: Got next segment. Inline: false Jun 26 06:07:34.199342823 (68h56m34.082756451s) [upload=55149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:07:34.199349731 (68h56m34.08276335s) [upload=55149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:07:34.391855411 (68h56m34.275269037s) [seg_pos={0 0}, upload=55149]: Begin upload segment... Jun 26 06:07:34.391904587 (68h56m34.275318203s) [seg_pos={0 0}, upload=55149]: Done begin upload segment. Jun 26 06:07:34.391941863 (68h56m34.275355496s) [noise=true , address=104.36.82.2:4732 , piece_id=PR7UO3K4PXEW2JP76V4XCS6AY6TBM2LEO7HNMQQET6R6C4XIZEXA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.392454984 (68h56m34.275868613s) [noise=true , address=104.36.82.10:4079 , piece_id=TB2DQYCLMYCGW4AH7UUYA76LH7HAZ7DQXORXLD3OFFUZ3K6YFZEQ, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.392553957 (68h56m34.275967628s) [noise=true , address=77.90.31.18:4805 , piece_id=XHKZ73DY6IVJMAH27ECE3DNKBDRQXJPJHPYQKCJ4TC5VSCW6JK7Q, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.392812908 (68h56m34.276226534s) [noise=true , address=104.36.82.2:4730 , piece_id=AEL4RMSLRP7YJAZI4IQK4GU4DIZ7RMQKSYNPZ5PRT4IYHR5PJ7IA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.392928787 (68h56m34.27634242s) [noise=true , address=104.36.82.8:4972 , piece_id=U274UCKWDQFE3GPOZZKUMM4YRVZTAZ42W3G4HGQFKZQ6XG7345OA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.393230901 (68h56m34.27664453s) [noise=true , address=77.90.31.12:4590 , piece_id=DOJIZT7TAWAHOX6CF7IX2WEJB6XJEFBRTDX7QYEIMC2W22CUV4RQ, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.393255302 (68h56m34.276668953s) [noise=true , address=104.36.82.7:4912 , piece_id=VMUXS5UTJLKKI3CWETZLBEYDD6VSHYWBEKWKFZBSER3IZXQRISWA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.393573218 (68h56m34.276986863s) [noise=true , address=45.91.116.11:4117 , piece_id=NKY222CCKL6BQJKFPLJCW3MGVXS3YWIL5RGF5JMRKOVNMHNNIN6A, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.393654244 (68h56m34.277067874s) [noise=true , address=104.36.82.11:4115 , piece_id=Y34VUOSKDFJMFRBU3PM5MBHP5S6IRZ4UCJRN2NMMDKJNVLGWHFJQ, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.393896341 (68h56m34.27730997s) [noise=true , address=104.36.82.5:4837 , piece_id=COIKZOO6JB5F2CKZZ7LWQMZBY7J3STETSKP4ZGAZJGZXWQTXU5OA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.394043971 (68h56m34.277457598s) [noise=true , address=104.36.82.14:4223 , piece_id=MN46FCXJVH3RVDMZWDHOHTYMSSV2TMSPIY3GGHUJ5KN3AWO6QNPA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.394403654 (68h56m34.277817291s) [noise=true , address=45.91.116.10:4083 , piece_id=Z3QK33GCLPA2UBA3CYJCQWITJC37BECNJ5ES3YDQAFCLXCZQBI2A, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.394899130 (68h56m34.278312759s) [noise=true , address=104.36.82.16:4294 , piece_id=7B52P5Z6R7WHOEENSK2YNIXXIJUPAJKFD6E2RBERBHTEJUPE6WXQ, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.395282379 (68h56m34.278696009s) [noise=true , address=104.36.82.13:4187 , piece_id=IWC7ZUK7PLBWSTXSIFN7HUE5MJMHB5KIXYVVWG5BSSXTBJOQT6MQ, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.395610250 (68h56m34.279023882s) [noise=true , address=45.91.116.7:4009 , piece_id=CGRYJDA3SUR7X7I4FK7EH3J4D5D4S5NA723LYUVVNPYDOHV3Y2CQ, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.396229442 (68h56m34.279643055s) [noise=true , address=104.36.82.12:4149 , piece_id=7WJQY3RVLWC5IYIYALIOUEKNY3VFQT3EL3G4ZCZVEL54M7MBQVHA, seg_pos={0 0}, upload=55149]: Uploading piece... Jun 26 06:07:34.994340732 (68h56m34.877754366s) [upload=55149]: Next returned nil segment Jun 26 06:07:34.994364237 (68h56m34.877777873s) [upload=55149]: Waiting for error group managing segments... Jun 26 06:07:35.131003229 (68h56m35.014416856s) [noise=true , address=104.36.82.7:4912 , piece_id=VMUXS5UTJLKKI3CWETZLBEYDD6VSHYWBEKWKFZBSER3IZXQRISWA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.132758799 (68h56m35.016172412s) [noise=true , address=104.36.82.13:4187 , piece_id=IWC7ZUK7PLBWSTXSIFN7HUE5MJMHB5KIXYVVWG5BSSXTBJOQT6MQ, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.136553572 (68h56m35.019967199s) [noise=true , address=104.36.82.2:4732 , piece_id=PR7UO3K4PXEW2JP76V4XCS6AY6TBM2LEO7HNMQQET6R6C4XIZEXA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.139873863 (68h56m35.023287481s) [noise=true , address=104.36.82.2:4730 , piece_id=AEL4RMSLRP7YJAZI4IQK4GU4DIZ7RMQKSYNPZ5PRT4IYHR5PJ7IA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.140506287 (68h56m35.023919908s) [noise=true , address=104.36.82.8:4972 , piece_id=U274UCKWDQFE3GPOZZKUMM4YRVZTAZ42W3G4HGQFKZQ6XG7345OA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.142002530 (68h56m35.025416149s) [noise=true , address=104.36.82.11:4115 , piece_id=Y34VUOSKDFJMFRBU3PM5MBHP5S6IRZ4UCJRN2NMMDKJNVLGWHFJQ, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.148499245 (68h56m35.031912872s) [noise=true , address=104.36.82.16:4294 , piece_id=7B52P5Z6R7WHOEENSK2YNIXXIJUPAJKFD6E2RBERBHTEJUPE6WXQ, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.159206327 (68h56m35.042619957s) [noise=true , address=104.36.82.14:4223 , piece_id=MN46FCXJVH3RVDMZWDHOHTYMSSV2TMSPIY3GGHUJ5KN3AWO6QNPA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.168035797 (68h56m35.051449435s) [noise=true , address=104.36.82.10:4079 , piece_id=TB2DQYCLMYCGW4AH7UUYA76LH7HAZ7DQXORXLD3OFFUZ3K6YFZEQ, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:35.222934520 (68h56m35.106348136s) [noise=true , address=104.36.82.12:4149 , piece_id=7WJQY3RVLWC5IYIYALIOUEKNY3VFQT3EL3G4ZCZVEL54M7MBQVHA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:07:36.075834575 (68h56m35.959248191s) [noise=true , address=104.36.82.5:4837 , piece_id=COIKZOO6JB5F2CKZZ7LWQMZBY7J3STETSKP4ZGAZJGZXWQTXU5OA, seg_pos={0 0}, upload=55149]: Done uploading piece. err: <nil> Jun 26 06:08:03.910982931 (68h57m3.794396567s) [noise=true , address=45.91.116.10:4083 , piece_id=Z3QK33GCLPA2UBA3CYJCQWITJC37BECNJ5ES3YDQAFCLXCZQBI2A, seg_pos={0 0}, upload=55149]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: rpc: dial tcp 45.91.116.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:03.911100610 (68h57m3.794514231s) [noise=true , address=45.91.116.7:4009 , piece_id=CGRYJDA3SUR7X7I4FK7EH3J4D5D4S5NA723LYUVVNPYDOHV3Y2CQ, seg_pos={0 0}, upload=55149]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 139.59.144.22:56752->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:03.911286078 (68h57m3.794699714s) [noise=true , address=77.90.31.12:4590 , piece_id=DOJIZT7TAWAHOX6CF7IX2WEJB6XJEFBRTDX7QYEIMC2W22CUV4RQ, seg_pos={0 0}, upload=55149]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:38514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:03.911400969 (68h57m3.794814597s) [noise=true , address=45.91.116.11:4117 , piece_id=NKY222CCKL6BQJKFPLJCW3MGVXS3YWIL5RGF5JMRKOVNMHNNIN6A, seg_pos={0 0}, upload=55149]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:03.911452986 (68h57m3.7948666s) [noise=true , address=77.90.31.18:4805 , piece_id=XHKZ73DY6IVJMAH27ECE3DNKBDRQXJPJHPYQKCJ4TC5VSCW6JK7Q, seg_pos={0 0}, upload=55149]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 139.59.144.22:33912->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:03.911525488 (68h57m3.794939126s) [seg_pos={0 0}, upload=55149]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 26 06:08:04.016849292 (68h57m3.900262909s) [upload=55149]: Done uploading segments... Jun 26 06:08:04.016899796 (68h57m3.900313423s) [upload=55149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.108345963s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00105152s FAIL (2025-06-26 06:07:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001080511s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.05358866s FAIL (2025-06-26 00:45:16 UTC) orbiter warning: context deadline exceeded Duration: 1.053679065s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 941.915434ms FAIL (2025-06-26 00:44:34 UTC) orbiter warning: context deadline exceeded Duration: 941.947553ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 965.408374ms FAIL (2025-06-26 00:44:15 UTC) orbiter warning: context deadline exceeded Duration: 965.44107ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.186983419s FAIL (2025-06-26 00:43:58 UTC) orbiter warning: context deadline exceeded Duration: 1.187016119s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.108715233s FAIL (2025-06-26 00:43:12 UTC) orbiter warning: context deadline exceeded Duration: 1.108751243s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.034271823s FAIL (2025-06-26 00:42:52 UTC) orbiter warning: context deadline exceeded Duration: 1.034305185s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 964.676384ms FAIL (2025-06-26 00:42:25 UTC) orbiter warning: context deadline exceeded Duration: 964.709179ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.037827328s FAIL (2025-06-26 00:41:46 UTC) orbiter warning: context deadline exceeded Duration: 1.037859958s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 966.152136ms FAIL (2025-06-26 00:40:36 UTC) orbiter warning: context deadline exceeded Duration: 966.200727ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.448931189s FAIL (2025-06-26 00:40:04 UTC) orbiter warning: context deadline exceeded Duration: 1.448963543s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 874.195965ms FAIL (2025-06-26 00:39:15 UTC) orbiter warning: context deadline exceeded Duration: 874.242049ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.168492563s FAIL (2025-06-26 00:38:28 UTC) orbiter warning: context deadline exceeded Duration: 1.168527633s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)