Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 406352 reports (Page 5187 of 8128)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.......X..........................] duration: 2.818946351s FAIL (2025-08-31 15:44:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819001136s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 3.053153916s FAIL (2025-08-31 15:44:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.053179812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX...XX.X....................] duration: 3.356161652s FAIL (2025-08-31 15:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.356202074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX...XX.X...................] duration: 3.17122586s FAIL (2025-08-31 15:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.171329569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX....X.......................] duration: 2.802581179s FAIL (2025-08-31 15:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.80261222s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.......X.........................] duration: 2.941409661s FAIL (2025-08-31 15:44:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.941458488s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 2.952741361s FAIL (2025-08-31 15:43:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.952768326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.......X........................] duration: 2.733395453s FAIL (2025-08-31 15:43:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.733422103s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX....X......................] duration: 2.471195836s FAIL (2025-08-31 15:43:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.471245222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration: 3.195084919s FAIL (2025-08-31 15:43:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.195122031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXX...XX.X..................] duration: 5.605123738s FAIL (2025-08-31 15:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.605263145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 2.999193472s FAIL (2025-08-31 15:42:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.999216661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.......X.......................] duration: 2.877919883s FAIL (2025-08-31 15:42:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.877937826s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 3.015221001s FAIL (2025-08-31 15:42:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.01525159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX....X.....................] duration: 2.614697538s FAIL (2025-08-31 15:42:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.614727257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXX...XX.X.................] duration: 5.679843887s FAIL (2025-08-31 15:42:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.679873566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XX....X....................] duration: 2.342405158s FAIL (2025-08-31 15:41:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.34243663s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.......X......................] duration: 2.785718979s FAIL (2025-08-31 15:41:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.785739088s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XXX...XX.X................] duration: 5.944604621s FAIL (2025-08-31 15:41:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.94463605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 2.815614273s FAIL (2025-08-31 15:41:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.815652689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XXX...XX.X...............] duration: 3.783993107s FAIL (2025-08-31 15:41:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.784041694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.......X.....................] duration: 2.563954001s FAIL (2025-08-31 15:41:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.564067764s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XX....X...................] duration: 3.386475822s FAIL (2025-08-31 15:41:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.386512369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.......X....................] duration: 2.880346742s FAIL (2025-08-31 15:40:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.880389782s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX....X..................] duration: 2.513594838s FAIL (2025-08-31 15:40:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.513651643s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 3.442385652s FAIL (2025-08-31 15:40:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442435588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XXX...XX.X..............] duration: 3.372679326s FAIL (2025-08-31 15:40:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.372716896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XX....X.................] duration: 2.873149716s FAIL (2025-08-31 15:39:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.87318031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XXX...XX.X.............] duration: 3.908591998s FAIL (2025-08-31 15:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.908625165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.......X...................] duration: 3.037727248s FAIL (2025-08-31 15:39:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.037748444s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 3.208285554s FAIL (2025-08-31 15:39:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.208315623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 2.975507183s FAIL (2025-08-31 15:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.975595665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXX...XX.X............] duration: 3.337029072s FAIL (2025-08-31 15:39:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.337153961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 3.223784401s FAIL (2025-08-31 15:39:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.223817671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XX....X................] duration: 2.705115768s FAIL (2025-08-31 15:38:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.705156888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXX...XX.X...........] duration: 5.625715494s FAIL (2025-08-31 15:38:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.625818924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.......X..................] duration: 2.783895267s FAIL (2025-08-31 15:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.783917917s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XX....X...............] duration: 2.832782415s FAIL (2025-08-31 15:38:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.832815312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXX...XX.X..........] duration: 5.650461655s FAIL (2025-08-31 15:38:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.650496902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 3.191480065s FAIL (2025-08-31 15:38:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.191522961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.......X.................] duration: 3.229174276s FAIL (2025-08-31 15:38:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.229199865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XX....X..............] duration: 2.680093014s FAIL (2025-08-31 15:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.680117957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXX...XX.X.........] duration: 3.222657868s FAIL (2025-08-31 15:37:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.222688282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XX....X.............] duration: 3.925128423s FAIL (2025-08-31 15:37:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.925178548s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.......X................] duration: 2.902793598s FAIL (2025-08-31 15:37:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.90282198s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 3.0820032s FAIL (2025-08-31 15:37:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.082058052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXX...XX.X........] duration: 5.382803376s FAIL (2025-08-31 15:37:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.38283773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.......X...............] duration: 2.789038322s FAIL (2025-08-31 15:36:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789074052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XX....X............] duration: 2.393919802s FAIL (2025-08-31 15:36:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.393946564s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.......X..............] duration: 2.745575986s FAIL (2025-08-31 15:36:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.745599454s edge gateway us-select-1

1
...
5185
5186
5187
5188
5189
...
8128