Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.475156508s FAIL (2025-09-22 05:46:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.47518912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X....X...XX...XX....XX..............X...X........X........] duration: 2.941337999s FAIL (2025-09-22 05:46:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.941359464s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.566301838s FAIL (2025-09-22 05:46:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.566334103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X.XX..X.............XX.........................] duration: 2.438652021s FAIL (2025-09-22 05:45:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.438685198s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X....X...XX...XX....XX..............X...X........X.......] duration: 3.100617118s FAIL (2025-09-22 05:45:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.100639632s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.609189128s FAIL (2025-09-22 05:45:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.609223331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X....X...XX...XX....XX..............X...X........X......] duration: 2.698618779s FAIL (2025-09-22 05:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.698665221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X...X.XX..X.............XX........................] duration: 2.680168603s FAIL (2025-09-22 05:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.680224341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..........X...X.XX..X.............XX.......................] duration: 2.440723748s FAIL (2025-09-22 05:44:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.440752898s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X....X...XX...XX....XX..............X...X........X.....] duration: 2.900073107s FAIL (2025-09-22 05:44:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.900185575s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.433839836s FAIL (2025-09-22 05:44:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.433884759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..........X...X.XX..X.............XX......................] duration: 2.697235785s FAIL (2025-09-22 05:44:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.697264074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.672865288s FAIL (2025-09-22 05:43:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.672917575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X....X...XX...XX....XX..............X...X........X....] duration: 2.665827732s FAIL (2025-09-22 05:43:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.665849816s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X....X...XX...XX....XX..............X...X........X...] duration: 3.163180881s FAIL (2025-09-22 05:43:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.163238947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.432579078s FAIL (2025-09-22 05:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.432604914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........X....X...XX...XX....XX..............X...X........X..] duration: 2.892063679s FAIL (2025-09-22 05:43:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.892097324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..........X...X.XX..X.............XX.....................] duration: 2.481380169s FAIL (2025-09-22 05:42:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.481428867s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........X....X...XX...XX....XX..............X...X........X.] duration: 2.768381649s FAIL (2025-09-22 05:42:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.768407919s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........X....X...XX...XX....XX..............X...X........X] duration: 30.00168078s FAIL (2025-09-22 05:42:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001718495s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.571128088s FAIL (2025-09-22 05:42:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.571162742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..........X...X.XX..X.............XX....................] duration: 2.531881789s FAIL (2025-09-22 05:42:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.531915587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.436348854s FAIL (2025-09-22 05:41:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.43638682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.528567394s FAIL (2025-09-22 05:41:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.528596353s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X....X...XX...XX....XX..............X...X........] duration: 2.682137736s FAIL (2025-09-22 05:41:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.682245372s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..........X...X.XX..X.............XX...................] duration: 2.34080991s FAIL (2025-09-22 05:41:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.340852412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.504462697s FAIL (2025-09-22 05:41:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.504494853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..........X...X.XX..X.............XX..................] duration: 2.410278134s FAIL (2025-09-22 05:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.410305495s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............X....X...XX...XX....XX..............X...X.......] duration: 2.80482643s FAIL (2025-09-22 05:40:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.804867897s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.824527051s FAIL (2025-09-22 05:40:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.824578137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X..........X...X.XX..X.............XX.................] duration: 2.607122263s FAIL (2025-09-22 05:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.607158625s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............X....X...XX...XX....XX..............X...X......] duration: 3.17711069s FAIL (2025-09-22 05:40:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.177145591s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............X....X...XX...XX....XX..............X...X.....] duration: 2.829411891s FAIL (2025-09-22 05:39:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829451324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.376905848s FAIL (2025-09-22 05:39:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.376935237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.504789919s FAIL (2025-09-22 05:39:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.504825024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X..........X...X.XX..X.............XX................] duration: 2.486607777s FAIL (2025-09-22 05:39:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.486642517s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..............X....X...XX...XX....XX..............X...X....] duration: 2.639698489s FAIL (2025-09-22 05:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.639728137s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.631515451s FAIL (2025-09-22 05:38:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.631557137s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..............X....X...XX...XX....XX..............X...X...] duration: 2.720035264s FAIL (2025-09-22 05:38:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.720089203s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X..........X...X.XX..X.............XX...............] duration: 2.451308598s FAIL (2025-09-22 05:37:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.451564019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.498719231s FAIL (2025-09-22 05:37:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.498750605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..............X....X...XX...XX....XX..............X...X..] duration: 2.886006365s FAIL (2025-09-22 05:37:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.886053029s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X..........X...X.XX..X.............XX..............] duration: 2.598552244s FAIL (2025-09-22 05:37:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.598581801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X..........X...X.XX..X.............XX.............] duration: 2.344963556s FAIL (2025-09-22 05:36:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.344991981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.63369495s FAIL (2025-09-22 05:36:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.633729054s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X..............X....X...XX...XX....XX..............X...X.] duration: 2.89713408s FAIL (2025-09-22 05:36:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.897164489s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X..........X...X.XX..X.............XX............] duration: 2.709904254s FAIL (2025-09-22 05:36:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.709936285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..............X....X...XX...XX....XX..............X...X] duration: 30.000903754s FAIL (2025-09-22 05:36:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000983683s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X..........X...X.XX..X.............XX...........] duration: 2.853033299s FAIL (2025-09-22 05:36:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.853076928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.478524846s FAIL (2025-09-22 05:35:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.478576133s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)