Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X........X.X..............................X..........X...X] duration: 30.110337249s FAIL (2025-09-22 05:13:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:12:52.523477769 (567h5m13.015910235s) [upload=498863]: Starting upload Sep 22 05:12:52.523581725 (567h5m13.016014175s) [upload=498863]: Done starting upload Sep 22 05:12:52.523707123 (567h5m13.016139556s) [upload=498863]: Uploading segments... Sep 22 05:12:52.523991347 (567h5m13.01642381s) [upload=498863]: Got next segment. Inline: false Sep 22 05:12:52.524075179 (567h5m13.01650763s) [upload=498863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:12:52.524094821 (567h5m13.016527282s) [upload=498863]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:12:52.636839862 (567h5m13.129272316s) [seg_pos={0 0}, upload=498863]: Begin upload segment... Sep 22 05:12:52.636918319 (567h5m13.129350759s) [seg_pos={0 0}, upload=498863]: Done begin upload segment. Sep 22 05:12:52.636952839 (567h5m13.129385281s) [noise=true , address=45.91.116.11:4117 , piece_id=63J6DYPINT6TU75XTD4S7EZ7SVNVKKPADERJ3K4463QHCRO7GPPQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.636983865 (567h5m13.129416314s) [noise=true , address=145.79.227.16:4909 , piece_id=Z5WGVCFFQ6GUXEIBQXKOZQDZM3E5RRPXXE6BLSBA7LVODAWWVLYA, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.637311966 (567h5m13.129744395s) [noise=true , address=45.91.116.14:4189 , piece_id=JLEHWHBNZTKTYM4OAWCY44TMZYLNEG5A7KGXMCABB5Y677VPK42Q, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.637521319 (567h5m13.129953771s) [noise=true , address=77.90.31.14:4663 , piece_id=KLHWWAPPYZXKDJJ7U3PGGQ4PGBFFP35TLTQOCBRKXIOEDR3LCMZQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.642153061 (567h5m13.134585513s) [noise=true , address=77.90.31.18:4805 , piece_id=JHBXQ6UJN3GN5YHF6ODWWKCY7DDNNDEL3HCXBAJHAJVPYBJYHYGQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.642630539 (567h5m13.135062991s) [noise=true , address=45.91.116.7:4009 , piece_id=HPJJTRLI43FLOMJCVQ4R2YT6F6WQBWTCZV3HPE4EHR7MFLYJC5BA, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.642976263 (567h5m13.135408711s) [noise=true , address=45.91.116.4:4938 , piece_id=KZ5Q2APHJV2IPHP6YEN5DDSHNUG7ZZVQBMXVRO3RUNZ7YFJXN4UA, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.643444818 (567h5m13.135877269s) [noise=true , address=145.79.227.14:4837 , piece_id=5WPV4K3J6YFQZH3W3VR4LGZOZLRHGCE6FRMGIKBL7KHYPN2UKT7A, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.644040569 (567h5m13.136473022s) [noise=true , address=77.90.31.10:4518 , piece_id=AJMBNZMOL7SHZYJ5AGQZOYMMNEMJSY4TGU7XWF55QD2GDI2X3GKQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.644599934 (567h5m13.137032387s) [noise=true , address=45.91.116.13:4155 , piece_id=SH5XSK6E6E4J6DSIRDHLTTI3LDBLGC4RQN5DH3KUNTULCXYSK7CA, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.645088026 (567h5m13.137520473s) [noise=true , address=77.90.31.11:4554 , piece_id=6XYEK6IVGFJ34VQGCVJLL2VXFQPR3ZX3YYR6HQ4Y5SL3DZDZFRLQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.645564133 (567h5m13.137996586s) [noise=true , address=145.79.227.15:4874 , piece_id=OW4NLLPLNHKBZ4QAABZH7FS2OKH5L3A536MMZ6ZKGGTMTQJK37NA, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.646066535 (567h5m13.138498988s) [noise=true , address=45.91.116.4:4940 , piece_id=55AISF6FOPT7MNNLIAABGE4DV63WINFTF3W2OTEBOZ5I6IYIS4EQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.646573918 (567h5m13.139006371s) [noise=true , address=145.79.227.3:4151 , piece_id=5HCE6GHWKBSQR6SSKECA5QNU4Y23OVZES6KWFAIHAQVZG4HUHDQQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.647069400 (567h5m13.139501849s) [noise=true , address=77.90.31.17:4771 , piece_id=OXDGHHWYZHXRYAC3P4TPDKMRTQU3DTRDYGRWK7QQZYGZCEG7YZUQ, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:52.647555683 (567h5m13.139988132s) [noise=true , address=77.90.31.13:4628 , piece_id=SE7UJEWLCBGM3N6OCXLSV655MSVBH4R66VMN4IJFOV2LXOMHU6QA, seg_pos={0 0}, upload=498863]: Uploading piece... Sep 22 05:12:53.263869405 (567h5m13.756301841s) [upload=498863]: Next returned nil segment Sep 22 05:12:53.263880756 (567h5m13.7563132s) [upload=498863]: Waiting for error group managing segments... Sep 22 05:12:53.385550938 (567h5m13.877983374s) [noise=true , address=45.91.116.14:4189 , piece_id=JLEHWHBNZTKTYM4OAWCY44TMZYLNEG5A7KGXMCABB5Y677VPK42Q, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.385772818 (567h5m13.878205252s) [noise=true , address=45.91.116.4:4938 , piece_id=KZ5Q2APHJV2IPHP6YEN5DDSHNUG7ZZVQBMXVRO3RUNZ7YFJXN4UA, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.388794490 (567h5m13.881226918s) [noise=true , address=77.90.31.14:4663 , piece_id=KLHWWAPPYZXKDJJ7U3PGGQ4PGBFFP35TLTQOCBRKXIOEDR3LCMZQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.399230502 (567h5m13.891662936s) [noise=true , address=45.91.116.7:4009 , piece_id=HPJJTRLI43FLOMJCVQ4R2YT6F6WQBWTCZV3HPE4EHR7MFLYJC5BA, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.399805338 (567h5m13.892237775s) [noise=true , address=45.91.116.11:4117 , piece_id=63J6DYPINT6TU75XTD4S7EZ7SVNVKKPADERJ3K4463QHCRO7GPPQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.401048161 (567h5m13.89348059s) [noise=true , address=45.91.116.13:4155 , piece_id=SH5XSK6E6E4J6DSIRDHLTTI3LDBLGC4RQN5DH3KUNTULCXYSK7CA, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.403099157 (567h5m13.895531593s) [noise=true , address=45.91.116.4:4940 , piece_id=55AISF6FOPT7MNNLIAABGE4DV63WINFTF3W2OTEBOZ5I6IYIS4EQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.404472930 (567h5m13.896905366s) [noise=true , address=77.90.31.10:4518 , piece_id=AJMBNZMOL7SHZYJ5AGQZOYMMNEMJSY4TGU7XWF55QD2GDI2X3GKQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.411272720 (567h5m13.903705172s) [noise=true , address=77.90.31.11:4554 , piece_id=6XYEK6IVGFJ34VQGCVJLL2VXFQPR3ZX3YYR6HQ4Y5SL3DZDZFRLQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.412105971 (567h5m13.904538406s) [noise=true , address=77.90.31.18:4805 , piece_id=JHBXQ6UJN3GN5YHF6ODWWKCY7DDNNDEL3HCXBAJHAJVPYBJYHYGQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.429011790 (567h5m13.921444233s) [noise=true , address=77.90.31.17:4771 , piece_id=OXDGHHWYZHXRYAC3P4TPDKMRTQU3DTRDYGRWK7QQZYGZCEG7YZUQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:53.429505453 (567h5m13.921937887s) [noise=true , address=77.90.31.13:4628 , piece_id=SE7UJEWLCBGM3N6OCXLSV655MSVBH4R66VMN4IJFOV2LXOMHU6QA, seg_pos={0 0}, upload=498863]: Done uploading piece. err: <nil> Sep 22 05:12:55.059778231 (567h5m15.552210674s) [noise=true , address=145.79.227.16:4909 , piece_id=Z5WGVCFFQ6GUXEIBQXKOZQDZM3E5RRPXXE6BLSBA7LVODAWWVLYA, seg_pos={0 0}, upload=498863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:22.242576479 (567h5m42.735008921s) [noise=true , address=145.79.227.15:4874 , piece_id=OW4NLLPLNHKBZ4QAABZH7FS2OKH5L3A536MMZ6ZKGGTMTQJK37NA, seg_pos={0 0}, upload=498863]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:13:22.242618475 (567h5m42.73505091s) [noise=true , address=145.79.227.3:4151 , piece_id=5HCE6GHWKBSQR6SSKECA5QNU4Y23OVZES6KWFAIHAQVZG4HUHDQQ, seg_pos={0 0}, upload=498863]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; context deadline exceeded Sep 22 05:13:22.242696858 (567h5m42.735129302s) [noise=true , address=145.79.227.14:4837 , piece_id=5WPV4K3J6YFQZH3W3VR4LGZOZLRHGCE6FRMGIKBL7KHYPN2UKT7A, seg_pos={0 0}, upload=498863]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:13:22.242746074 (567h5m42.735178527s) [seg_pos={0 0}, upload=498863]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:13:22.350894871 (567h5m42.843327317s) [upload=498863]: Done uploading segments... Sep 22 05:13:22.350927195 (567h5m42.843359644s) [upload=498863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.110368369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X.X........X.XX......X....X..............X....X...XX.] duration: 2.786791178s FAIL (2025-09-22 05:12:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786813351s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.423377297s FAIL (2025-09-22 05:12:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.423420066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........X.X........X.XX......X....X..............X....X...XX] duration: 30.001135723s FAIL (2025-09-22 05:12:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001172374s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X........X.X..............................X..........X...] duration: 2.657689394s FAIL (2025-09-22 05:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:10:20.160829023 (567h2m40.653261472s) [upload=498831]: Starting upload Sep 22 05:10:20.160951803 (567h2m40.653384247s) [upload=498831]: Done starting upload Sep 22 05:10:20.161124042 (567h2m40.653556487s) [upload=498831]: Uploading segments... Sep 22 05:10:20.161393912 (567h2m40.653826341s) [upload=498831]: Got next segment. Inline: false Sep 22 05:10:20.161478809 (567h2m40.653911262s) [upload=498831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:10:20.161487216 (567h2m40.65391965s) [upload=498831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:10:20.273192985 (567h2m40.765625542s) [seg_pos={0 0}, upload=498831]: Begin upload segment... Sep 22 05:10:20.273277477 (567h2m40.765709928s) [seg_pos={0 0}, upload=498831]: Done begin upload segment. Sep 22 05:10:20.273347066 (567h2m40.765779518s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.273981720 (567h2m40.766414174s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274088061 (567h2m40.766520512s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274471571 (567h2m40.766904005s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274499027 (567h2m40.766931476s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274768423 (567h2m40.767200855s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274992416 (567h2m40.767424917s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275057033 (567h2m40.767489465s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275470045 (567h2m40.767902476s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275544762 (567h2m40.767977216s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275766390 (567h2m40.768198833s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275983970 (567h2m40.768416422s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276060943 (567h2m40.768493395s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276411086 (567h2m40.768843539s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276848504 (567h2m40.769280955s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.283288222 (567h2m40.775720651s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.897454373 (567h2m41.389886815s) [upload=498831]: Next returned nil segment Sep 22 05:10:20.897480705 (567h2m41.389913147s) [upload=498831]: Waiting for error group managing segments... Sep 22 05:10:21.036537067 (567h2m41.528969502s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.036596187 (567h2m41.529028619s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.043649090 (567h2m41.536081529s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.045147521 (567h2m41.537579955s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046161948 (567h2m41.538594392s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046936603 (567h2m41.539369037s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049332133 (567h2m41.54176458s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049638470 (567h2m41.542070906s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050008910 (567h2m41.542441344s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050140542 (567h2m41.542572977s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.053835484 (567h2m41.546267935s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.056398698 (567h2m41.548831144s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:22.509558692 (567h2m43.001991127s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:37.553873381 (567h2m58.046305815s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:49.873167614 (567h3m10.365600056s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873214826 (567h3m10.36564727s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873253554 (567h3m10.365686005s) [seg_pos={0 0}, upload=498831]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:10:49.979831378 (567h3m10.472263821s) [upload=498831]: Done uploading segments... Sep 22 05:10:49.979883041 (567h3m10.472315484s) [upload=498831]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.657723538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.573128437s FAIL (2025-09-22 05:11:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.573165544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........X.X..............................X..........X..] duration: 2.537303272s FAIL (2025-09-22 05:11:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:10:20.160829023 (567h2m40.653261472s) [upload=498831]: Starting upload Sep 22 05:10:20.160951803 (567h2m40.653384247s) [upload=498831]: Done starting upload Sep 22 05:10:20.161124042 (567h2m40.653556487s) [upload=498831]: Uploading segments... Sep 22 05:10:20.161393912 (567h2m40.653826341s) [upload=498831]: Got next segment. Inline: false Sep 22 05:10:20.161478809 (567h2m40.653911262s) [upload=498831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:10:20.161487216 (567h2m40.65391965s) [upload=498831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:10:20.273192985 (567h2m40.765625542s) [seg_pos={0 0}, upload=498831]: Begin upload segment... Sep 22 05:10:20.273277477 (567h2m40.765709928s) [seg_pos={0 0}, upload=498831]: Done begin upload segment. Sep 22 05:10:20.273347066 (567h2m40.765779518s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.273981720 (567h2m40.766414174s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274088061 (567h2m40.766520512s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274471571 (567h2m40.766904005s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274499027 (567h2m40.766931476s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274768423 (567h2m40.767200855s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274992416 (567h2m40.767424917s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275057033 (567h2m40.767489465s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275470045 (567h2m40.767902476s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275544762 (567h2m40.767977216s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275766390 (567h2m40.768198833s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275983970 (567h2m40.768416422s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276060943 (567h2m40.768493395s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276411086 (567h2m40.768843539s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276848504 (567h2m40.769280955s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.283288222 (567h2m40.775720651s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.897454373 (567h2m41.389886815s) [upload=498831]: Next returned nil segment Sep 22 05:10:20.897480705 (567h2m41.389913147s) [upload=498831]: Waiting for error group managing segments... Sep 22 05:10:21.036537067 (567h2m41.528969502s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.036596187 (567h2m41.529028619s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.043649090 (567h2m41.536081529s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.045147521 (567h2m41.537579955s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046161948 (567h2m41.538594392s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046936603 (567h2m41.539369037s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049332133 (567h2m41.54176458s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049638470 (567h2m41.542070906s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050008910 (567h2m41.542441344s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050140542 (567h2m41.542572977s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.053835484 (567h2m41.546267935s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.056398698 (567h2m41.548831144s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:22.509558692 (567h2m43.001991127s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:37.553873381 (567h2m58.046305815s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:49.873167614 (567h3m10.365600056s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873214826 (567h3m10.36564727s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873253554 (567h3m10.365686005s) [seg_pos={0 0}, upload=498831]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:10:49.979831378 (567h3m10.472263821s) [upload=498831]: Done uploading segments... Sep 22 05:10:49.979883041 (567h3m10.472315484s) [upload=498831]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.537347589s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X.X........X.XX......X....X..............X....X...X] duration: 30.000890611s FAIL (2025-09-22 05:11:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000920178s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X........X.X..............................X..........X.] duration: 2.708737574s FAIL (2025-09-22 05:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:10:20.160829023 (567h2m40.653261472s) [upload=498831]: Starting upload Sep 22 05:10:20.160951803 (567h2m40.653384247s) [upload=498831]: Done starting upload Sep 22 05:10:20.161124042 (567h2m40.653556487s) [upload=498831]: Uploading segments... Sep 22 05:10:20.161393912 (567h2m40.653826341s) [upload=498831]: Got next segment. Inline: false Sep 22 05:10:20.161478809 (567h2m40.653911262s) [upload=498831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:10:20.161487216 (567h2m40.65391965s) [upload=498831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:10:20.273192985 (567h2m40.765625542s) [seg_pos={0 0}, upload=498831]: Begin upload segment... Sep 22 05:10:20.273277477 (567h2m40.765709928s) [seg_pos={0 0}, upload=498831]: Done begin upload segment. Sep 22 05:10:20.273347066 (567h2m40.765779518s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.273981720 (567h2m40.766414174s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274088061 (567h2m40.766520512s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274471571 (567h2m40.766904005s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274499027 (567h2m40.766931476s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274768423 (567h2m40.767200855s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274992416 (567h2m40.767424917s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275057033 (567h2m40.767489465s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275470045 (567h2m40.767902476s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275544762 (567h2m40.767977216s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275766390 (567h2m40.768198833s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275983970 (567h2m40.768416422s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276060943 (567h2m40.768493395s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276411086 (567h2m40.768843539s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276848504 (567h2m40.769280955s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.283288222 (567h2m40.775720651s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.897454373 (567h2m41.389886815s) [upload=498831]: Next returned nil segment Sep 22 05:10:20.897480705 (567h2m41.389913147s) [upload=498831]: Waiting for error group managing segments... Sep 22 05:10:21.036537067 (567h2m41.528969502s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.036596187 (567h2m41.529028619s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.043649090 (567h2m41.536081529s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.045147521 (567h2m41.537579955s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046161948 (567h2m41.538594392s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046936603 (567h2m41.539369037s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049332133 (567h2m41.54176458s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049638470 (567h2m41.542070906s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050008910 (567h2m41.542441344s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050140542 (567h2m41.542572977s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.053835484 (567h2m41.546267935s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.056398698 (567h2m41.548831144s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:22.509558692 (567h2m43.001991127s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:37.553873381 (567h2m58.046305815s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:49.873167614 (567h3m10.365600056s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873214826 (567h3m10.36564727s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873253554 (567h3m10.365686005s) [seg_pos={0 0}, upload=498831]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:10:49.979831378 (567h3m10.472263821s) [upload=498831]: Done uploading segments... Sep 22 05:10:49.979883041 (567h3m10.472315484s) [upload=498831]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.708831785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X........X.X..............................X..........X] duration: 30.108496599s FAIL (2025-09-22 05:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:10:20.160829023 (567h2m40.653261472s) [upload=498831]: Starting upload Sep 22 05:10:20.160951803 (567h2m40.653384247s) [upload=498831]: Done starting upload Sep 22 05:10:20.161124042 (567h2m40.653556487s) [upload=498831]: Uploading segments... Sep 22 05:10:20.161393912 (567h2m40.653826341s) [upload=498831]: Got next segment. Inline: false Sep 22 05:10:20.161478809 (567h2m40.653911262s) [upload=498831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:10:20.161487216 (567h2m40.65391965s) [upload=498831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:10:20.273192985 (567h2m40.765625542s) [seg_pos={0 0}, upload=498831]: Begin upload segment... Sep 22 05:10:20.273277477 (567h2m40.765709928s) [seg_pos={0 0}, upload=498831]: Done begin upload segment. Sep 22 05:10:20.273347066 (567h2m40.765779518s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.273981720 (567h2m40.766414174s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274088061 (567h2m40.766520512s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274471571 (567h2m40.766904005s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274499027 (567h2m40.766931476s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274768423 (567h2m40.767200855s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.274992416 (567h2m40.767424917s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275057033 (567h2m40.767489465s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275470045 (567h2m40.767902476s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275544762 (567h2m40.767977216s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275766390 (567h2m40.768198833s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.275983970 (567h2m40.768416422s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276060943 (567h2m40.768493395s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276411086 (567h2m40.768843539s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.276848504 (567h2m40.769280955s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.283288222 (567h2m40.775720651s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Uploading piece... Sep 22 05:10:20.897454373 (567h2m41.389886815s) [upload=498831]: Next returned nil segment Sep 22 05:10:20.897480705 (567h2m41.389913147s) [upload=498831]: Waiting for error group managing segments... Sep 22 05:10:21.036537067 (567h2m41.528969502s) [noise=true , address=45.91.116.13:4154 , piece_id=H3Z4PSDGHLMHUCIIJPFDE5G5NOCDAFTOWJCM67ZAWMJAXIKFJXMA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.036596187 (567h2m41.529028619s) [noise=true , address=45.91.116.8:4047 , piece_id=4DXCPAV3USSENXWCYQZ4SGEQCJPVLR62PNE4EE7KMHACTE56Y72A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.043649090 (567h2m41.536081529s) [noise=true , address=77.90.31.12:4589 , piece_id=XGTYKD2LAZMXCI7OD3QWNCRE7GRZTI3H27CSTR7ZI6X4FZV3HGPQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.045147521 (567h2m41.537579955s) [noise=true , address=45.91.116.12:4225 , piece_id=HMCP5R5WNEG2H35SCO2BP6RKFQT2AMJJY24NABRMUAXKVBRGJF3A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046161948 (567h2m41.538594392s) [noise=true , address=45.91.116.14:4190 , piece_id=EZLEG3VXG2NBCTMLFAWRBMG22QK5X5HNWWNXPGYS32HJW6V64IIA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.046936603 (567h2m41.539369037s) [noise=true , address=45.91.116.10:4081 , piece_id=TZZJH542S4J66S6DRUX7OBYAARK65ELXOARFFRHB2RQA67YUNBKQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049332133 (567h2m41.54176458s) [noise=true , address=45.91.116.14:4189 , piece_id=2H2CROCHCP32SJIBFJKZYWCY23WJKJQ2FQV6GLCGVTE2YYGGURGA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.049638470 (567h2m41.542070906s) [noise=true , address=45.91.116.4:4937 , piece_id=MVAHP65HIKJQNV6KLP7HHNGFBFZ2QOCWJMUI3GQI7TJS5HI2E3LA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050008910 (567h2m41.542441344s) [noise=true , address=45.91.116.9:4263 , piece_id=KGY7VZLE5TFYMLWGOT7DLCDDENHXQ3FHQVPCN6DRBK3XKLVYEE4Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.050140542 (567h2m41.542572977s) [noise=true , address=45.91.116.7:4010 , piece_id=35XUSX5GTIRF76NFNGRHB5JYIDL53A4CM62ORDB4PMKTKGRHNU6A, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.053835484 (567h2m41.546267935s) [noise=true , address=77.90.31.16:4734 , piece_id=PN447ZEQ6MC66D5CNRKF37NEYF3ZFVYS3GTWGQTZ5775FXP34WHA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:21.056398698 (567h2m41.548831144s) [noise=true , address=77.90.31.10:4520 , piece_id=UBIJXTRV3WHT6KMTDHR2Y7CMII4YZBWKFC2KCPQ5LY2X2RR4L6HQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: <nil> Sep 22 05:10:22.509558692 (567h2m43.001991127s) [noise=true , address=145.79.227.18:4005 , piece_id=2G3WLG3HEBDSYFKKZQELET6BKHSDLFUZG5L64DC75I57JZ4IJ64Q, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:37.553873381 (567h2m58.046305815s) [noise=true , address=145.79.227.11:4732 , piece_id=VD4GGWKRPVWQCYFSUQOPHXYVS5TVFCTTIYXJM7RLCHGYYDGJP7MA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:10:49.873167614 (567h3m10.365600056s) [noise=true , address=145.79.227.19:4077 , piece_id=LXV5U2ZWY6QWPW23V7GVAEUM6KRUAMG6YNAN4GC4QSD7TNDXCKJQ, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873214826 (567h3m10.36564727s) [noise=true , address=145.79.227.17:4970 , piece_id=GXUMYZPCJKPG6HHEWSIDGWTJ62KB262V5CHJDUSMIV362F35VFWA, seg_pos={0 0}, upload=498831]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:10:49.873253554 (567h3m10.365686005s) [seg_pos={0 0}, upload=498831]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:10:49.979831378 (567h3m10.472263821s) [upload=498831]: Done uploading segments... Sep 22 05:10:49.979883041 (567h3m10.472315484s) [upload=498831]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.108515447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.636781487s FAIL (2025-09-22 05:10:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.636838519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X.X........X.XX......X....X..............X....X...] duration: 3.187857856s FAIL (2025-09-22 05:10:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.187884801s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.433713603s FAIL (2025-09-22 05:10:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.433778313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X.X........X.XX......X....X..............X....X..] duration: 2.893652671s FAIL (2025-09-22 05:09:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.893702888s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X.X........X.XX......X....X..............X....X.] duration: 2.810449638s FAIL (2025-09-22 05:09:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.810482872s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X........X.X..............................X..........] duration: 2.498194215s FAIL (2025-09-22 05:09:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.498228493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.601379832s FAIL (2025-09-22 05:09:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.601422636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X.X........X.XX......X....X..............X....X] duration: 30.001011464s FAIL (2025-09-22 05:09:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001040837s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.70501848s FAIL (2025-09-22 05:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.705060633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X........X.X..............................X.........] duration: 2.41845281s FAIL (2025-09-22 05:08:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.418502254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.621245054s FAIL (2025-09-22 05:08:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.621284686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X........X.X..............................X........] duration: 2.724777678s FAIL (2025-09-22 05:07:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.724816433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.563564993s FAIL (2025-09-22 05:07:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.563598053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.X........X.XX......X....X..............X....] duration: 2.917370317s FAIL (2025-09-22 05:07:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.917403846s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X........X.X..............................X.......] duration: 2.863826679s FAIL (2025-09-22 05:07:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.863909855s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..............X.X........X.XX......X....X..............X...] duration: 2.916733824s FAIL (2025-09-22 05:07:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.916767042s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X........X.X..............................X......] duration: 2.547868565s FAIL (2025-09-22 05:06:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.547907304s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..............X.X........X.XX......X....X..............X..] duration: 2.738963278s FAIL (2025-09-22 05:06:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.739054242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.614876386s FAIL (2025-09-22 05:06:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.61489904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.117928741s FAIL (2025-09-22 05:06:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.117986819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..............X.X........X.XX......X....X..............X.] duration: 2.836846433s FAIL (2025-09-22 05:06:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.836885141s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X........X.X..............................X.....] duration: 2.50337189s FAIL (2025-09-22 05:05:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.503421305s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..............X.X........X.XX......X....X..............X] duration: 30.000515292s FAIL (2025-09-22 05:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000538682s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X........X.X..............................X....] duration: 2.713584344s FAIL (2025-09-22 05:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.713676467s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..............X.X........X.XX......X....X..............] duration: 2.762781896s FAIL (2025-09-22 05:04:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.762808612s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.............X........X.X..............................X...] duration: 2.686905573s FAIL (2025-09-22 05:04:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.686983885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.............X........X.X..............................X..] duration: 2.477000793s FAIL (2025-09-22 05:03:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.477058343s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..............X.X........X.XX......X....X.............] duration: 2.725193239s FAIL (2025-09-22 05:02:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.725221553s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X.............X........X.X..............................X.] duration: 2.425876469s FAIL (2025-09-22 05:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.425907174s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X..............X.X........X.XX......X....X............] duration: 2.773676587s FAIL (2025-09-22 05:02:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.773707716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.X.............X........X.X..............................X] duration: 30.11169501s FAIL (2025-09-22 05:01:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:01:18.280703479 (566h53m38.773135926s) [upload=498701]: Starting upload Sep 22 05:01:18.280818725 (566h53m38.773251174s) [upload=498701]: Done starting upload Sep 22 05:01:18.280941997 (566h53m38.773374437s) [upload=498701]: Uploading segments... Sep 22 05:01:18.281248039 (566h53m38.773680485s) [upload=498701]: Got next segment. Inline: false Sep 22 05:01:18.281322276 (566h53m38.773754714s) [upload=498701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:01:18.281328293 (566h53m38.773760735s) [upload=498701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:01:18.389866657 (566h53m38.882299103s) [seg_pos={0 0}, upload=498701]: Begin upload segment... Sep 22 05:01:18.389968637 (566h53m38.882401085s) [seg_pos={0 0}, upload=498701]: Done begin upload segment. Sep 22 05:01:18.390003079 (566h53m38.88243552s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390060366 (566h53m38.882492815s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390592985 (566h53m38.883025449s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.390607819 (566h53m38.883040271s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391060923 (566h53m38.883493376s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391249051 (566h53m38.8836815s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391601968 (566h53m38.884034419s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.391666542 (566h53m38.884098985s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392001318 (566h53m38.884433772s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392035069 (566h53m38.884467522s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392392560 (566h53m38.884825012s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392478367 (566h53m38.884910811s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392790267 (566h53m38.885222722s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.392897662 (566h53m38.885330111s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.395892742 (566h53m38.888325188s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:18.396338103 (566h53m38.888770539s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Uploading piece... Sep 22 05:01:19.007076878 (566h53m39.499509314s) [upload=498701]: Next returned nil segment Sep 22 05:01:19.007290773 (566h53m39.49972322s) [upload=498701]: Waiting for error group managing segments... Sep 22 05:01:19.144508273 (566h53m39.636940707s) [noise=true , address=45.91.116.12:4225 , piece_id=2IS23PU2GB4RKESFAYJMSPGWMU7SXF4NJNZMIPPZKORCRA6RI2OQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.144755440 (566h53m39.637187897s) [noise=true , address=45.91.116.14:4191 , piece_id=QUO4OKPUY4VVHVYUIMOFUEGOCELZFJKIMMAUKLUMDE2FOTBGEOAQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.145455669 (566h53m39.637888102s) [noise=true , address=77.90.31.14:4661 , piece_id=RIBCGMRL4M34QM4S5FBT7MTYR636X23XEWVHR4UJRWELPWBDF7HQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.146439873 (566h53m39.638872324s) [noise=true , address=45.91.116.13:4153 , piece_id=QYNSX74HHQCM6HNGTSLWGXDWKPNMGEQHQMRCRZWXEPUDCWBD3FYA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.147796984 (566h53m39.640229419s) [noise=true , address=77.90.31.10:4517 , piece_id=XBWFLRDITS2A3JUWNV4GCT3CQPCXNQKKGJ4GU623HPJN7CU2SG7A, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.148815008 (566h53m39.641247451s) [noise=true , address=45.91.116.8:4047 , piece_id=STVV4NMXJRU7FTETZOOTC37TP5HZWCJJCHZ4DFJ63P63YHHCWN6Q, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.149421157 (566h53m39.641853599s) [noise=true , address=45.91.116.10:4083 , piece_id=RTGQUKJFYN6TNGAXTOJEHESKGVVWTWN6VKPI4IYRI2M43KX3L2WA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.154621176 (566h53m39.647053638s) [noise=true , address=45.91.116.4:4937 , piece_id=NA4NIYMD6JGHUVHJV7KMBRDSHU5UZ33PMGN2OFNRRBRBNQ5Y3OVQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.155579694 (566h53m39.648012132s) [noise=true , address=45.91.116.10:4081 , piece_id=2UQXDKKGCY5WO7W53CWGQ7AURYICKR6JEWV56G76MICA4XXOX3ZA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.158069627 (566h53m39.650502071s) [noise=true , address=45.91.116.5:4297 , piece_id=LLEOSRQ3SOMNWBO67IJ2DVGGZ5VSRYX5XSPDCNQIJWDVNPAXNFAA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:19.159636451 (566h53m39.652068884s) [noise=true , address=77.90.31.12:4592 , piece_id=7SYNQAF3XWODZPIPDF7M4X2KF2KI42M2O5VEXNKHAIKBTPZ2KQRQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: <nil> Sep 22 05:01:48.001189478 (566h54m8.493621938s) [noise=true , address=145.79.227.5:4221 , piece_id=G4J3B7YDZM4FCJYK7WJ6D454KZQRWCMGSXRLBIT4O7TVCNAGHVVA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001276830 (566h54m8.49370928s) [noise=true , address=145.79.227.14:4840 , piece_id=T7JPNTK5FHT62TD2VHWMT6RZY5GZ2S2WEGSZ3GNK4QAW3WG7OGSA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001301932 (566h54m8.493734376s) [noise=true , address=145.79.227.2:4113 , piece_id=57DNHQWHENQAGKYXVQMYBQATAEWXDQYX3TD35CMKZ7P3UQBU5IKA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001396431 (566h54m8.49382886s) [noise=true , address=145.79.227.4:4186 , piece_id=DNJUW5NK2FYNULILODQCJICAWUKRJHOKE7ULTXJXYI5KF32J4VTQ, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:01:48.001440014 (566h54m8.49387246s) [noise=true , address=145.79.227.7:4295 , piece_id=J2DRN62XDIEENLLGFQYURUR7GUZYJCPB6UFG26R7E6XPZ5N4OGHA, seg_pos={0 0}, upload=498701]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:01:48.001469148 (566h54m8.4939016s) [seg_pos={0 0}, upload=498701]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:01:48.110561660 (566h54m8.602994103s) [upload=498701]: Done uploading segments... Sep 22 05:01:48.110590234 (566h54m8.603022678s) [upload=498701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.11173618s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X......X..............X.X........X.XX......X....X...........] duration: 2.797983878s FAIL (2025-09-22 05:01:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.798007567s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X......X..............X.X........X.XX......X....X..........] duration: 2.840474103s FAIL (2025-09-22 05:00:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.840514003s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X.............X........X.X..............................] duration: 3.044964537s FAIL (2025-09-22 05:00:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:37:32.968419254 (566h29m53.460851689s) [upload=498358]: Starting upload Sep 22 04:37:32.968513489 (566h29m53.460945923s) [upload=498358]: Done starting upload Sep 22 04:37:32.968809636 (566h29m53.461242073s) [upload=498358]: Uploading segments... Sep 22 04:37:32.969040319 (566h29m53.461472764s) [upload=498358]: Got next segment. Inline: false Sep 22 04:37:32.969163301 (566h29m53.461595748s) [upload=498358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:37:32.969172925 (566h29m53.461605372s) [upload=498358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:37:33.080710082 (566h29m53.573142539s) [seg_pos={0 0}, upload=498358]: Begin upload segment... Sep 22 04:37:33.080779723 (566h29m53.573212156s) [seg_pos={0 0}, upload=498358]: Done begin upload segment. Sep 22 04:37:33.080813398 (566h29m53.573245854s) [noise=true , address=45.91.116.5:4299 , piece_id=25UQC76R26MMVM5FTIIZ54UYQFIZFRGFD5CMCAEVWZOUVHZG4DMQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.080845445 (566h29m53.57327789s) [noise=true , address=45.91.116.13:4155 , piece_id=Y3Y5EPNU34SK7ORARWWYOB7N56U43MFLH5CZOIVJONJ2HSHKKJLA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081381456 (566h29m53.573813942s) [noise=true , address=45.91.116.4:4937 , piece_id=T5EIJ4LVXH2YAY7HFFEF5YKWSXPVUK2BD6VCXAYVSWBH7IU7YH3Q, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081398382 (566h29m53.573830837s) [noise=true , address=45.91.116.10:4081 , piece_id=Z3V3FFVACDB6JFLKZS24YXY5A2AJ7HPIAWKETEGWK4QOT5SNCBGQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081837215 (566h29m53.574269729s) [noise=true , address=45.91.116.4:4940 , piece_id=UXIF6WUZ2PJIKMPWY7TIQ3VKZXARAVN53HCSXYUI635EOWOWKFHQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081945290 (566h29m53.574377735s) [noise=true , address=77.90.31.18:4807 , piece_id=LPOTW6ZJJULK3OA6XOTP2KXZOBYFXS34M2ZAHLWGP2MJA2JB4HLQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.082577692 (566h29m53.57501014s) [noise=true , address=77.90.31.11:4553 , piece_id=O2EWRCDMBKGRY2FDJDHM7F5OF555SBX55FM76B7NTC2BHS5DGEFQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.086324936 (566h29m53.578757379s) [noise=true , address=145.79.227.7:4293 , piece_id=BEVQOKI2OZQSY3HIA4CNISMNCBEGY24UJUVMHQGSZ5ADWAHWSROQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.086657904 (566h29m53.579090358s) [noise=true , address=45.91.116.6:4975 , piece_id=FLFCY5VZG4MTLRNMJHYWJEE3KVSHXH3JJCH5OAQ3YK6FI53MLRYA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.087163160 (566h29m53.579595606s) [noise=true , address=145.79.227.4:4187 , piece_id=UXPTQXBW64WFA4NU6UBX27ORZX2G42MT4LYTCJZIHKH554EE4PLA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.087616272 (566h29m53.580048714s) [noise=true , address=145.79.227.13:4804 , piece_id=633SWUJBABGZ6BB2X7SF4KERVZUZ3XPDOLK4TQPI7WRCNUP27X6Q, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088013469 (566h29m53.580445911s) [noise=true , address=45.91.116.14:4191 , piece_id=FSNAAUR6PL7P7EFLFF6HIDZE2BTMZZCW7PSQYO2CVUN7XFCKSGJQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088434498 (566h29m53.580866944s) [noise=true , address=45.91.116.13:4153 , piece_id=TBGNBL43AWH3P6WUZBBTMAEGML5INHON47ELYNTBRMVISN5IM7ZQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088856358 (566h29m53.58128881s) [noise=true , address=145.79.227.18:4005 , piece_id=N6EUH4XX5UTTT7PY5OJ33PGXUFXJ6J4OUNEOMBK7HQ3GWKU6GNCA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.089265407 (566h29m53.581697864s) [noise=true , address=77.90.31.13:4627 , piece_id=7MO6XXZAXKVBRVCCWHOXRWZSZQXNRBXJVWBBLZBDCTMPYJDYY6KQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.089873098 (566h29m53.582305553s) [noise=true , address=45.91.116.9:4261 , piece_id=ONNMC5GGO2KY5VQKFAO4IDCQC5W4EYYHIZBXPTG5EFIFQJ5KSTVQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.708120441 (566h29m54.200552898s) [upload=498358]: Next returned nil segment Sep 22 04:37:33.708153651 (566h29m54.200586107s) [upload=498358]: Waiting for error group managing segments... Sep 22 04:37:33.839160529 (566h29m54.331592974s) [noise=true , address=45.91.116.13:4155 , piece_id=Y3Y5EPNU34SK7ORARWWYOB7N56U43MFLH5CZOIVJONJ2HSHKKJLA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.839427891 (566h29m54.331860345s) [noise=true , address=77.90.31.18:4807 , piece_id=LPOTW6ZJJULK3OA6XOTP2KXZOBYFXS34M2ZAHLWGP2MJA2JB4HLQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.846774756 (566h29m54.33920721s) [noise=true , address=77.90.31.11:4553 , piece_id=O2EWRCDMBKGRY2FDJDHM7F5OF555SBX55FM76B7NTC2BHS5DGEFQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.847735851 (566h29m54.340168295s) [noise=true , address=45.91.116.4:4940 , piece_id=UXIF6WUZ2PJIKMPWY7TIQ3VKZXARAVN53HCSXYUI635EOWOWKFHQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.848490229 (566h29m54.340922678s) [noise=true , address=45.91.116.5:4299 , piece_id=25UQC76R26MMVM5FTIIZ54UYQFIZFRGFD5CMCAEVWZOUVHZG4DMQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.849277942 (566h29m54.341710379s) [noise=true , address=45.91.116.4:4937 , piece_id=T5EIJ4LVXH2YAY7HFFEF5YKWSXPVUK2BD6VCXAYVSWBH7IU7YH3Q, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.849971905 (566h29m54.342404352s) [noise=true , address=45.91.116.14:4191 , piece_id=FSNAAUR6PL7P7EFLFF6HIDZE2BTMZZCW7PSQYO2CVUN7XFCKSGJQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.850312853 (566h29m54.342745289s) [noise=true , address=45.91.116.6:4975 , piece_id=FLFCY5VZG4MTLRNMJHYWJEE3KVSHXH3JJCH5OAQ3YK6FI53MLRYA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.857758135 (566h29m54.350190579s) [noise=true , address=45.91.116.13:4153 , piece_id=TBGNBL43AWH3P6WUZBBTMAEGML5INHON47ELYNTBRMVISN5IM7ZQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.859398765 (566h29m54.351831211s) [noise=true , address=77.90.31.13:4627 , piece_id=7MO6XXZAXKVBRVCCWHOXRWZSZQXNRBXJVWBBLZBDCTMPYJDYY6KQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.862640583 (566h29m54.35507303s) [noise=true , address=45.91.116.9:4261 , piece_id=ONNMC5GGO2KY5VQKFAO4IDCQC5W4EYYHIZBXPTG5EFIFQJ5KSTVQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.865251417 (566h29m54.357683864s) [noise=true , address=45.91.116.10:4081 , piece_id=Z3V3FFVACDB6JFLKZS24YXY5A2AJ7HPIAWKETEGWK4QOT5SNCBGQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:35.575020212 (566h29m56.067452669s) [noise=true , address=145.79.227.18:4005 , piece_id=N6EUH4XX5UTTT7PY5OJ33PGXUFXJ6J4OUNEOMBK7HQ3GWKU6GNCA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:38:02.694641552 (566h30m23.187073992s) [noise=true , address=145.79.227.7:4293 , piece_id=BEVQOKI2OZQSY3HIA4CNISMNCBEGY24UJUVMHQGSZ5ADWAHWSROQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694933020 (566h30m23.18736546s) [noise=true , address=145.79.227.13:4804 , piece_id=633SWUJBABGZ6BB2X7SF4KERVZUZ3XPDOLK4TQPI7WRCNUP27X6Q, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694953564 (566h30m23.18738602s) [noise=true , address=145.79.227.4:4187 , piece_id=UXPTQXBW64WFA4NU6UBX27ORZX2G42MT4LYTCJZIHKH554EE4PLA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694998543 (566h30m23.187430994s) [seg_pos={0 0}, upload=498358]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:38:02.800453269 (566h30m23.292885745s) [upload=498358]: Done uploading segments... Sep 22 04:38:02.800523687 (566h30m23.292956145s) [upload=498358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.045004835s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X......X..............X.X........X.XX......X....X.........] duration: 2.758673799s FAIL (2025-09-22 04:59:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.758753298s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X......X..............X.X........X.XX......X....X........] duration: 2.773444654s FAIL (2025-09-22 04:59:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.77347279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X.X.............X........X.X.............................] duration: 2.462224847s FAIL (2025-09-22 04:59:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:37:32.968419254 (566h29m53.460851689s) [upload=498358]: Starting upload Sep 22 04:37:32.968513489 (566h29m53.460945923s) [upload=498358]: Done starting upload Sep 22 04:37:32.968809636 (566h29m53.461242073s) [upload=498358]: Uploading segments... Sep 22 04:37:32.969040319 (566h29m53.461472764s) [upload=498358]: Got next segment. Inline: false Sep 22 04:37:32.969163301 (566h29m53.461595748s) [upload=498358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:37:32.969172925 (566h29m53.461605372s) [upload=498358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:37:33.080710082 (566h29m53.573142539s) [seg_pos={0 0}, upload=498358]: Begin upload segment... Sep 22 04:37:33.080779723 (566h29m53.573212156s) [seg_pos={0 0}, upload=498358]: Done begin upload segment. Sep 22 04:37:33.080813398 (566h29m53.573245854s) [noise=true , address=45.91.116.5:4299 , piece_id=25UQC76R26MMVM5FTIIZ54UYQFIZFRGFD5CMCAEVWZOUVHZG4DMQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.080845445 (566h29m53.57327789s) [noise=true , address=45.91.116.13:4155 , piece_id=Y3Y5EPNU34SK7ORARWWYOB7N56U43MFLH5CZOIVJONJ2HSHKKJLA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081381456 (566h29m53.573813942s) [noise=true , address=45.91.116.4:4937 , piece_id=T5EIJ4LVXH2YAY7HFFEF5YKWSXPVUK2BD6VCXAYVSWBH7IU7YH3Q, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081398382 (566h29m53.573830837s) [noise=true , address=45.91.116.10:4081 , piece_id=Z3V3FFVACDB6JFLKZS24YXY5A2AJ7HPIAWKETEGWK4QOT5SNCBGQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081837215 (566h29m53.574269729s) [noise=true , address=45.91.116.4:4940 , piece_id=UXIF6WUZ2PJIKMPWY7TIQ3VKZXARAVN53HCSXYUI635EOWOWKFHQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081945290 (566h29m53.574377735s) [noise=true , address=77.90.31.18:4807 , piece_id=LPOTW6ZJJULK3OA6XOTP2KXZOBYFXS34M2ZAHLWGP2MJA2JB4HLQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.082577692 (566h29m53.57501014s) [noise=true , address=77.90.31.11:4553 , piece_id=O2EWRCDMBKGRY2FDJDHM7F5OF555SBX55FM76B7NTC2BHS5DGEFQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.086324936 (566h29m53.578757379s) [noise=true , address=145.79.227.7:4293 , piece_id=BEVQOKI2OZQSY3HIA4CNISMNCBEGY24UJUVMHQGSZ5ADWAHWSROQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.086657904 (566h29m53.579090358s) [noise=true , address=45.91.116.6:4975 , piece_id=FLFCY5VZG4MTLRNMJHYWJEE3KVSHXH3JJCH5OAQ3YK6FI53MLRYA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.087163160 (566h29m53.579595606s) [noise=true , address=145.79.227.4:4187 , piece_id=UXPTQXBW64WFA4NU6UBX27ORZX2G42MT4LYTCJZIHKH554EE4PLA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.087616272 (566h29m53.580048714s) [noise=true , address=145.79.227.13:4804 , piece_id=633SWUJBABGZ6BB2X7SF4KERVZUZ3XPDOLK4TQPI7WRCNUP27X6Q, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088013469 (566h29m53.580445911s) [noise=true , address=45.91.116.14:4191 , piece_id=FSNAAUR6PL7P7EFLFF6HIDZE2BTMZZCW7PSQYO2CVUN7XFCKSGJQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088434498 (566h29m53.580866944s) [noise=true , address=45.91.116.13:4153 , piece_id=TBGNBL43AWH3P6WUZBBTMAEGML5INHON47ELYNTBRMVISN5IM7ZQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088856358 (566h29m53.58128881s) [noise=true , address=145.79.227.18:4005 , piece_id=N6EUH4XX5UTTT7PY5OJ33PGXUFXJ6J4OUNEOMBK7HQ3GWKU6GNCA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.089265407 (566h29m53.581697864s) [noise=true , address=77.90.31.13:4627 , piece_id=7MO6XXZAXKVBRVCCWHOXRWZSZQXNRBXJVWBBLZBDCTMPYJDYY6KQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.089873098 (566h29m53.582305553s) [noise=true , address=45.91.116.9:4261 , piece_id=ONNMC5GGO2KY5VQKFAO4IDCQC5W4EYYHIZBXPTG5EFIFQJ5KSTVQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.708120441 (566h29m54.200552898s) [upload=498358]: Next returned nil segment Sep 22 04:37:33.708153651 (566h29m54.200586107s) [upload=498358]: Waiting for error group managing segments... Sep 22 04:37:33.839160529 (566h29m54.331592974s) [noise=true , address=45.91.116.13:4155 , piece_id=Y3Y5EPNU34SK7ORARWWYOB7N56U43MFLH5CZOIVJONJ2HSHKKJLA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.839427891 (566h29m54.331860345s) [noise=true , address=77.90.31.18:4807 , piece_id=LPOTW6ZJJULK3OA6XOTP2KXZOBYFXS34M2ZAHLWGP2MJA2JB4HLQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.846774756 (566h29m54.33920721s) [noise=true , address=77.90.31.11:4553 , piece_id=O2EWRCDMBKGRY2FDJDHM7F5OF555SBX55FM76B7NTC2BHS5DGEFQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.847735851 (566h29m54.340168295s) [noise=true , address=45.91.116.4:4940 , piece_id=UXIF6WUZ2PJIKMPWY7TIQ3VKZXARAVN53HCSXYUI635EOWOWKFHQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.848490229 (566h29m54.340922678s) [noise=true , address=45.91.116.5:4299 , piece_id=25UQC76R26MMVM5FTIIZ54UYQFIZFRGFD5CMCAEVWZOUVHZG4DMQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.849277942 (566h29m54.341710379s) [noise=true , address=45.91.116.4:4937 , piece_id=T5EIJ4LVXH2YAY7HFFEF5YKWSXPVUK2BD6VCXAYVSWBH7IU7YH3Q, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.849971905 (566h29m54.342404352s) [noise=true , address=45.91.116.14:4191 , piece_id=FSNAAUR6PL7P7EFLFF6HIDZE2BTMZZCW7PSQYO2CVUN7XFCKSGJQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.850312853 (566h29m54.342745289s) [noise=true , address=45.91.116.6:4975 , piece_id=FLFCY5VZG4MTLRNMJHYWJEE3KVSHXH3JJCH5OAQ3YK6FI53MLRYA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.857758135 (566h29m54.350190579s) [noise=true , address=45.91.116.13:4153 , piece_id=TBGNBL43AWH3P6WUZBBTMAEGML5INHON47ELYNTBRMVISN5IM7ZQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.859398765 (566h29m54.351831211s) [noise=true , address=77.90.31.13:4627 , piece_id=7MO6XXZAXKVBRVCCWHOXRWZSZQXNRBXJVWBBLZBDCTMPYJDYY6KQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.862640583 (566h29m54.35507303s) [noise=true , address=45.91.116.9:4261 , piece_id=ONNMC5GGO2KY5VQKFAO4IDCQC5W4EYYHIZBXPTG5EFIFQJ5KSTVQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.865251417 (566h29m54.357683864s) [noise=true , address=45.91.116.10:4081 , piece_id=Z3V3FFVACDB6JFLKZS24YXY5A2AJ7HPIAWKETEGWK4QOT5SNCBGQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:35.575020212 (566h29m56.067452669s) [noise=true , address=145.79.227.18:4005 , piece_id=N6EUH4XX5UTTT7PY5OJ33PGXUFXJ6J4OUNEOMBK7HQ3GWKU6GNCA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:38:02.694641552 (566h30m23.187073992s) [noise=true , address=145.79.227.7:4293 , piece_id=BEVQOKI2OZQSY3HIA4CNISMNCBEGY24UJUVMHQGSZ5ADWAHWSROQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694933020 (566h30m23.18736546s) [noise=true , address=145.79.227.13:4804 , piece_id=633SWUJBABGZ6BB2X7SF4KERVZUZ3XPDOLK4TQPI7WRCNUP27X6Q, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694953564 (566h30m23.18738602s) [noise=true , address=145.79.227.4:4187 , piece_id=UXPTQXBW64WFA4NU6UBX27ORZX2G42MT4LYTCJZIHKH554EE4PLA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694998543 (566h30m23.187430994s) [seg_pos={0 0}, upload=498358]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:38:02.800453269 (566h30m23.292885745s) [upload=498358]: Done uploading segments... Sep 22 04:38:02.800523687 (566h30m23.292956145s) [upload=498358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.46226015s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..X......X..............X.X........X.XX......X....X.......] duration: 2.788789449s FAIL (2025-09-22 04:58:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.788892852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X.X.............X........X.X............................] duration: 2.575968887s FAIL (2025-09-22 04:58:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:37:32.968419254 (566h29m53.460851689s) [upload=498358]: Starting upload Sep 22 04:37:32.968513489 (566h29m53.460945923s) [upload=498358]: Done starting upload Sep 22 04:37:32.968809636 (566h29m53.461242073s) [upload=498358]: Uploading segments... Sep 22 04:37:32.969040319 (566h29m53.461472764s) [upload=498358]: Got next segment. Inline: false Sep 22 04:37:32.969163301 (566h29m53.461595748s) [upload=498358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:37:32.969172925 (566h29m53.461605372s) [upload=498358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:37:33.080710082 (566h29m53.573142539s) [seg_pos={0 0}, upload=498358]: Begin upload segment... Sep 22 04:37:33.080779723 (566h29m53.573212156s) [seg_pos={0 0}, upload=498358]: Done begin upload segment. Sep 22 04:37:33.080813398 (566h29m53.573245854s) [noise=true , address=45.91.116.5:4299 , piece_id=25UQC76R26MMVM5FTIIZ54UYQFIZFRGFD5CMCAEVWZOUVHZG4DMQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.080845445 (566h29m53.57327789s) [noise=true , address=45.91.116.13:4155 , piece_id=Y3Y5EPNU34SK7ORARWWYOB7N56U43MFLH5CZOIVJONJ2HSHKKJLA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081381456 (566h29m53.573813942s) [noise=true , address=45.91.116.4:4937 , piece_id=T5EIJ4LVXH2YAY7HFFEF5YKWSXPVUK2BD6VCXAYVSWBH7IU7YH3Q, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081398382 (566h29m53.573830837s) [noise=true , address=45.91.116.10:4081 , piece_id=Z3V3FFVACDB6JFLKZS24YXY5A2AJ7HPIAWKETEGWK4QOT5SNCBGQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081837215 (566h29m53.574269729s) [noise=true , address=45.91.116.4:4940 , piece_id=UXIF6WUZ2PJIKMPWY7TIQ3VKZXARAVN53HCSXYUI635EOWOWKFHQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.081945290 (566h29m53.574377735s) [noise=true , address=77.90.31.18:4807 , piece_id=LPOTW6ZJJULK3OA6XOTP2KXZOBYFXS34M2ZAHLWGP2MJA2JB4HLQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.082577692 (566h29m53.57501014s) [noise=true , address=77.90.31.11:4553 , piece_id=O2EWRCDMBKGRY2FDJDHM7F5OF555SBX55FM76B7NTC2BHS5DGEFQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.086324936 (566h29m53.578757379s) [noise=true , address=145.79.227.7:4293 , piece_id=BEVQOKI2OZQSY3HIA4CNISMNCBEGY24UJUVMHQGSZ5ADWAHWSROQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.086657904 (566h29m53.579090358s) [noise=true , address=45.91.116.6:4975 , piece_id=FLFCY5VZG4MTLRNMJHYWJEE3KVSHXH3JJCH5OAQ3YK6FI53MLRYA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.087163160 (566h29m53.579595606s) [noise=true , address=145.79.227.4:4187 , piece_id=UXPTQXBW64WFA4NU6UBX27ORZX2G42MT4LYTCJZIHKH554EE4PLA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.087616272 (566h29m53.580048714s) [noise=true , address=145.79.227.13:4804 , piece_id=633SWUJBABGZ6BB2X7SF4KERVZUZ3XPDOLK4TQPI7WRCNUP27X6Q, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088013469 (566h29m53.580445911s) [noise=true , address=45.91.116.14:4191 , piece_id=FSNAAUR6PL7P7EFLFF6HIDZE2BTMZZCW7PSQYO2CVUN7XFCKSGJQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088434498 (566h29m53.580866944s) [noise=true , address=45.91.116.13:4153 , piece_id=TBGNBL43AWH3P6WUZBBTMAEGML5INHON47ELYNTBRMVISN5IM7ZQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.088856358 (566h29m53.58128881s) [noise=true , address=145.79.227.18:4005 , piece_id=N6EUH4XX5UTTT7PY5OJ33PGXUFXJ6J4OUNEOMBK7HQ3GWKU6GNCA, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.089265407 (566h29m53.581697864s) [noise=true , address=77.90.31.13:4627 , piece_id=7MO6XXZAXKVBRVCCWHOXRWZSZQXNRBXJVWBBLZBDCTMPYJDYY6KQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.089873098 (566h29m53.582305553s) [noise=true , address=45.91.116.9:4261 , piece_id=ONNMC5GGO2KY5VQKFAO4IDCQC5W4EYYHIZBXPTG5EFIFQJ5KSTVQ, seg_pos={0 0}, upload=498358]: Uploading piece... Sep 22 04:37:33.708120441 (566h29m54.200552898s) [upload=498358]: Next returned nil segment Sep 22 04:37:33.708153651 (566h29m54.200586107s) [upload=498358]: Waiting for error group managing segments... Sep 22 04:37:33.839160529 (566h29m54.331592974s) [noise=true , address=45.91.116.13:4155 , piece_id=Y3Y5EPNU34SK7ORARWWYOB7N56U43MFLH5CZOIVJONJ2HSHKKJLA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.839427891 (566h29m54.331860345s) [noise=true , address=77.90.31.18:4807 , piece_id=LPOTW6ZJJULK3OA6XOTP2KXZOBYFXS34M2ZAHLWGP2MJA2JB4HLQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.846774756 (566h29m54.33920721s) [noise=true , address=77.90.31.11:4553 , piece_id=O2EWRCDMBKGRY2FDJDHM7F5OF555SBX55FM76B7NTC2BHS5DGEFQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.847735851 (566h29m54.340168295s) [noise=true , address=45.91.116.4:4940 , piece_id=UXIF6WUZ2PJIKMPWY7TIQ3VKZXARAVN53HCSXYUI635EOWOWKFHQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.848490229 (566h29m54.340922678s) [noise=true , address=45.91.116.5:4299 , piece_id=25UQC76R26MMVM5FTIIZ54UYQFIZFRGFD5CMCAEVWZOUVHZG4DMQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.849277942 (566h29m54.341710379s) [noise=true , address=45.91.116.4:4937 , piece_id=T5EIJ4LVXH2YAY7HFFEF5YKWSXPVUK2BD6VCXAYVSWBH7IU7YH3Q, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.849971905 (566h29m54.342404352s) [noise=true , address=45.91.116.14:4191 , piece_id=FSNAAUR6PL7P7EFLFF6HIDZE2BTMZZCW7PSQYO2CVUN7XFCKSGJQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.850312853 (566h29m54.342745289s) [noise=true , address=45.91.116.6:4975 , piece_id=FLFCY5VZG4MTLRNMJHYWJEE3KVSHXH3JJCH5OAQ3YK6FI53MLRYA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.857758135 (566h29m54.350190579s) [noise=true , address=45.91.116.13:4153 , piece_id=TBGNBL43AWH3P6WUZBBTMAEGML5INHON47ELYNTBRMVISN5IM7ZQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.859398765 (566h29m54.351831211s) [noise=true , address=77.90.31.13:4627 , piece_id=7MO6XXZAXKVBRVCCWHOXRWZSZQXNRBXJVWBBLZBDCTMPYJDYY6KQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.862640583 (566h29m54.35507303s) [noise=true , address=45.91.116.9:4261 , piece_id=ONNMC5GGO2KY5VQKFAO4IDCQC5W4EYYHIZBXPTG5EFIFQJ5KSTVQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:33.865251417 (566h29m54.357683864s) [noise=true , address=45.91.116.10:4081 , piece_id=Z3V3FFVACDB6JFLKZS24YXY5A2AJ7HPIAWKETEGWK4QOT5SNCBGQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: <nil> Sep 22 04:37:35.575020212 (566h29m56.067452669s) [noise=true , address=145.79.227.18:4005 , piece_id=N6EUH4XX5UTTT7PY5OJ33PGXUFXJ6J4OUNEOMBK7HQ3GWKU6GNCA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:38:02.694641552 (566h30m23.187073992s) [noise=true , address=145.79.227.7:4293 , piece_id=BEVQOKI2OZQSY3HIA4CNISMNCBEGY24UJUVMHQGSZ5ADWAHWSROQ, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694933020 (566h30m23.18736546s) [noise=true , address=145.79.227.13:4804 , piece_id=633SWUJBABGZ6BB2X7SF4KERVZUZ3XPDOLK4TQPI7WRCNUP27X6Q, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694953564 (566h30m23.18738602s) [noise=true , address=145.79.227.4:4187 , piece_id=UXPTQXBW64WFA4NU6UBX27ORZX2G42MT4LYTCJZIHKH554EE4PLA, seg_pos={0 0}, upload=498358]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:38:02.694998543 (566h30m23.187430994s) [seg_pos={0 0}, upload=498358]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:38:02.800453269 (566h30m23.292885745s) [upload=498358]: Done uploading segments... Sep 22 04:38:02.800523687 (566h30m23.292956145s) [upload=498358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.576010493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X......X..............X.X........X.XX......X....X......] duration: 2.909117357s FAIL (2025-09-22 04:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.909142577s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)