Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 406347 reports (Page 5186 of 8127)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XXX...XX.X..................................] duration: 3.34695225s FAIL (2025-08-31 15:52:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.346982343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......X.....................................] duration: 2.920233988s FAIL (2025-08-31 15:52:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.920294221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XXX...XX.X.................................] duration: 2.770535784s FAIL (2025-08-31 15:52:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.770568731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......X....................................] duration: 2.890314103s FAIL (2025-08-31 15:52:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.890367113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XX....X.................................] duration: 2.598044088s FAIL (2025-08-31 15:52:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.598174636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.......X...................................] duration: 2.785496861s FAIL (2025-08-31 15:52:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.785520547s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................XXXX.....................................] duration: 2.810471203s FAIL (2025-08-31 15:52:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.810503149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................XXXX....................................] duration: 3.177091447s FAIL (2025-08-31 15:51:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.17719898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XX....X................................] duration: 2.345086717s FAIL (2025-08-31 15:51:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.345115936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XXX...XX.X................................] duration: 3.210086002s FAIL (2025-08-31 15:51:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.210119386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......X..................................] duration: 2.836894791s FAIL (2025-08-31 15:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.836918696s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XXX...XX.X...............................] duration: 8.345765837s FAIL (2025-08-31 15:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.345807031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......X.................................] duration: 2.984002899s FAIL (2025-08-31 15:51:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.984564353s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXXX...................................] duration: 2.916480843s FAIL (2025-08-31 15:50:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.916512178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XX....X...............................] duration: 2.480615255s FAIL (2025-08-31 15:50:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.480651844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XXX...XX.X..............................] duration: 3.088904895s FAIL (2025-08-31 15:50:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.088938036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......X................................] duration: 3.074301438s FAIL (2025-08-31 15:50:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.074344274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XXX...XX.X.............................] duration: 3.742045283s FAIL (2025-08-31 15:50:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.74208403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 3.716427505s FAIL (2025-08-31 15:49:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.716517806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XXX...XX.X............................] duration: 4.936874825s FAIL (2025-08-31 15:49:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.93693004s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.......X...............................] duration: 2.876832572s FAIL (2025-08-31 15:49:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.876864315s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XX....X..............................] duration: 2.734483918s FAIL (2025-08-31 15:49:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.734521922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XXX...XX.X...........................] duration: 4.721889072s FAIL (2025-08-31 15:49:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.721945975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XXX...XX.X..........................] duration: 3.187080353s FAIL (2025-08-31 15:48:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.187138963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 2.948775867s FAIL (2025-08-31 15:48:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.948815689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XX....X.............................] duration: 2.423755698s FAIL (2025-08-31 15:48:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.423800848s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.......X..............................] duration: 2.923660456s FAIL (2025-08-31 15:48:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.923705283s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX....X............................] duration: 2.766422572s FAIL (2025-08-31 15:48:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.766452185s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 3.629447744s FAIL (2025-08-31 15:47:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629475189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XXX...XX.X.........................] duration: 4.693994151s FAIL (2025-08-31 15:47:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.694033591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XXX...XX.X........................] duration: 3.139189645s FAIL (2025-08-31 15:47:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.139225282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.......X.............................] duration: 2.935544902s FAIL (2025-08-31 15:47:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.935590046s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX....X...........................] duration: 2.349059512s FAIL (2025-08-31 15:47:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.349095299s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 2.874083458s FAIL (2025-08-31 15:47:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.874108564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 3.027206374s FAIL (2025-08-31 15:46:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.027450496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XXX...XX.X.......................] duration: 4.255831481s FAIL (2025-08-31 15:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.25586217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.......X............................] duration: 2.786775447s FAIL (2025-08-31 15:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786801574s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 3.029974125s FAIL (2025-08-31 15:46:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.030058824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX....X..........................] duration: 3.625929942s FAIL (2025-08-31 15:46:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.626002359s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.......X...........................] duration: 2.793784095s FAIL (2025-08-31 15:45:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.793806555s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX...XX.X......................] duration: 7.566482802s FAIL (2025-08-31 15:45:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.566574219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 2.8771303s FAIL (2025-08-31 15:45:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.877408643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX....X.........................] duration: 3.730700489s FAIL (2025-08-31 15:45:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.730736784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX....X........................] duration: 2.492024116s FAIL (2025-08-31 15:45:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.492072605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX...XX.X.....................] duration: 4.558558618s FAIL (2025-08-31 15:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.558600517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.......X..........................] duration: 2.818946351s FAIL (2025-08-31 15:44:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819001136s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 3.053153916s FAIL (2025-08-31 15:44:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.053179812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX...XX.X....................] duration: 3.356161652s FAIL (2025-08-31 15:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.356202074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX...XX.X...................] duration: 3.17122586s FAIL (2025-08-31 15:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.171329569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX....X.......................] duration: 2.802581179s FAIL (2025-08-31 15:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.80261222s core uplink us-select-1

1
...
5184
5185
5186
5187
5188
...
8127