Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 406450 reports (Page 5187 of 8129)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 2.834050935s FAIL (2025-08-31 16:02:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.83407449s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......XXXX..................................................] duration: 3.462165194s FAIL (2025-08-31 16:02:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462250219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XX....X................................................] duration: 2.325409668s FAIL (2025-08-31 16:02:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.32545021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXX...XX.X...............................................] duration: 2.840528824s FAIL (2025-08-31 16:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.840563403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXX...XX.X..............................................] duration: 2.566075274s FAIL (2025-08-31 16:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.56610947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XX....X...............................................] duration: 2.464007677s FAIL (2025-08-31 16:02:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.464039077s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 3.102204039s FAIL (2025-08-31 16:01:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.102238499s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 3.052706115s FAIL (2025-08-31 16:01:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.05273624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XX....X..............................................] duration: 2.307283159s FAIL (2025-08-31 16:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.307314145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XXX...XX.X.............................................] duration: 3.648624354s FAIL (2025-08-31 16:01:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.648654421s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 2.769863708s FAIL (2025-08-31 16:01:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.770371916s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XXX...XX.X............................................] duration: 2.543952839s FAIL (2025-08-31 16:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.543992068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 3.994966392s FAIL (2025-08-31 16:00:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.995049159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XX....X.............................................] duration: 2.280476933s FAIL (2025-08-31 16:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.2805173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XX....X............................................] duration: 2.326330112s FAIL (2025-08-31 16:00:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.326363596s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......X..............................................] duration: 2.85832698s FAIL (2025-08-31 16:00:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.858359769s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 2.95433223s FAIL (2025-08-31 16:00:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.954359547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XXX...XX.X...........................................] duration: 2.860353594s FAIL (2025-08-31 15:59:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.860379162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XX....X...........................................] duration: 2.560678324s FAIL (2025-08-31 15:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.56070527s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.......X.............................................] duration: 2.736273563s FAIL (2025-08-31 15:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.736302327s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XXX...XX.X..........................................] duration: 3.409222843s FAIL (2025-08-31 15:59:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.409255637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XX....X..........................................] duration: 2.417132618s FAIL (2025-08-31 15:59:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.417164574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........XXXX..............................................] duration: 3.020463158s FAIL (2025-08-31 15:59:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.02048893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......X............................................] duration: 2.735946952s FAIL (2025-08-31 15:58:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.735973856s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XX....X.........................................] duration: 2.333540324s FAIL (2025-08-31 15:58:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.333567767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XXX...XX.X.........................................] duration: 3.372348847s FAIL (2025-08-31 15:58:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.372382367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XX....X........................................] duration: 5.577403103s FAIL (2025-08-31 15:58:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.577433944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 3.030542706s FAIL (2025-08-31 15:58:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.030570895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XX....X.......................................] duration: 2.447144003s FAIL (2025-08-31 15:58:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.447180947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XXX...XX.X........................................] duration: 5.430081701s FAIL (2025-08-31 15:57:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.430116062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.......X...........................................] duration: 3.107639278s FAIL (2025-08-31 15:57:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.107669024s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............XXXX............................................] duration: 2.805369966s FAIL (2025-08-31 15:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.805440809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XX....X......................................] duration: 2.458223797s FAIL (2025-08-31 15:57:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.45825316s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......X..........................................] duration: 2.918951525s FAIL (2025-08-31 15:57:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.918997613s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XXX...XX.X.......................................] duration: 5.035825279s FAIL (2025-08-31 15:57:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.035860354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............XXXX...........................................] duration: 3.063457053s FAIL (2025-08-31 15:56:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.063481926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......X.........................................] duration: 2.891621938s FAIL (2025-08-31 15:56:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.892030584s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XXX...XX.X......................................] duration: 3.720755071s FAIL (2025-08-31 15:56:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.720818816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XX....X.....................................] duration: 3.065911912s FAIL (2025-08-31 15:56:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.065944745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XXX...XX.X.....................................] duration: 5.091923164s FAIL (2025-08-31 15:56:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.091950865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............XXXX..........................................] duration: 2.916785604s FAIL (2025-08-31 15:55:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.916813916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......X........................................] duration: 2.884664219s FAIL (2025-08-31 15:55:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.884692552s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............XXXX.........................................] duration: 3.101440946s FAIL (2025-08-31 15:55:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.101468645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XX....X....................................] duration: 2.606089479s FAIL (2025-08-31 15:55:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.606127332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XXX...XX.X....................................] duration: 3.314308034s FAIL (2025-08-31 15:54:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.31433933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................XXXX........................................] duration: 3.341768678s FAIL (2025-08-31 15:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.341858847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XX....X...................................] duration: 3.375849319s FAIL (2025-08-31 15:54:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19m0.958033156s) [seg_pos={0 0}, upload=43339]: Begin upload segment... Aug 31 15:26:40.465652212 (49h19m0.958084658s) [seg_pos={0 0}, upload=43339]: Done begin upload segment. Aug 31 15:26:40.465704516 (49h19m0.958136963s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.465803704 (49h19m0.958236158s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466226430 (49h19m0.958658879s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466452323 (49h19m0.958884768s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466597073 (49h19m0.959029523s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466813077 (49h19m0.959245522s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.466999337 (49h19m0.959431783s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467175858 (49h19m0.959608301s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467395399 (49h19m0.959827852s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467458627 (49h19m0.959891069s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467794047 (49h19m0.960226496s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.467882165 (49h19m0.960314595s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468225046 (49h19m0.960657498s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468378846 (49h19m0.960811291s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.468641322 (49h19m0.961073771s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:40.469054985 (49h19m0.961487438s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Uploading piece... Aug 31 15:26:41.137614806 (49h19m1.630047243s) [upload=43339]: Next returned nil segment Aug 31 15:26:41.137633833 (49h19m1.630066281s) [upload=43339]: Waiting for error group managing segments... Aug 31 15:26:41.265108040 (49h19m1.757540495s) [noise=true , address=104.36.82.8:4970 , piece_id=IQSV7GQ45WNKEYIFX2UZ3W34GCBLYNMUILTAIOP4OWLO5MLKG53Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:41.283487416 (49h19m1.775919861s) [noise=true , address=104.36.82.13:4186 , piece_id=FYYRRLJYHCXCJOFWIZIC5YAAUWBBI6JY2WYE7WUSU76XUDHMMSVA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:58.049120704 (49h19m18.541553133s) [noise=true , address=77.90.31.18:4807 , piece_id=YX56D457WYFZLPUMUX3HCPKH3SKXYKF77PN76YFNEOPJOF6FSAUQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:26:59.808431274 (49h19m20.300863709s) [noise=true , address=45.91.116.5:4299 , piece_id=G2FGRTPUNCLIKMVXYYR6NVI3BX3B5SV3X53457WQIN2VVQZZY6JA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:05.931494704 (49h19m26.423927146s) [noise=true , address=77.90.31.16:4733 , piece_id=I3UI5YJKTGPQTB4ZLCWN3QQZCROZ37SQBI5543MURRNIPLZJQPIQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:08.536580583 (49h19m29.029013015s) [noise=true , address=45.91.116.12:4226 , piece_id=EGLPH6YR5LESWZWW6W4IYD4TYPXXUIHK744IAMGA66ZPDVUHCOLQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: <nil> Aug 31 15:27:09.909705539 (49h19m30.402137982s) [noise=true , address=77.90.31.15:4697 , piece_id=BDZ6BQYKY4VR4Q2AAID5U7JSRZWHILEMKPT7IHDZWIEEEVMJDQSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909785647 (49h19m30.402218084s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPZM7IUAY2KWVMVIG5N65IE4BFEXGIZ5ZBA4CW7RVGBBNHG7CO4Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.909861836 (49h19m30.402294271s) [noise=true , address=77.90.31.19:4842 , piece_id=75NYHJJOC5OVMHWR5MFDTWI7DFPT7KFHSZLVGYJO6SWJ7YJ26IFQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910107397 (49h19m30.40253984s) [noise=true , address=77.90.31.10:4517 , piece_id=VAH3T6USMSZZJQ26ZMUAUN5ZS7HAYLF52DYCAQO4UAU5CPSIR3WQ, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910262517 (49h19m30.402694961s) [noise=true , address=77.90.31.16:4735 , piece_id=J4XF33F3V546XFXBCQZHQKHZIIIJRRRD4JBHSR3SVQKAQTKR72ZA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Aug 31 15:27:09.910614447 (49h19m30.40304688s) [noise=true , address=77.90.31.13:4627 , piece_id=FYFLDA2ZGRRYC32HXOX2Y4S2JZZ3NIHT7Y2CTDPR3M4X3MFGKP2Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35866->77.90.31.13:4627: use of closed network connection Aug 31 15:27:09.910684130 (49h19m30.403116574s) [noise=true , address=45.91.116.7:4011 , piece_id=GA6EKFKDHRSELNKIIFODCO4P2UKX6G2DDYOT7MBJBHTBXUUGNQ5A, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55174->45.91.116.7:4011: use of closed network connection Aug 31 15:27:09.910866957 (49h19m30.4032994s) [noise=true , address=45.91.116.4:4937 , piece_id=Y3W2D3JRLEJBW5YNNB74PF6S3BIDHTJIJAT4NZXWQK23LD3YUCSA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40512->45.91.116.4:4937: use of closed network connection Aug 31 15:27:09.910935862 (49h19m30.403368295s) [noise=true , address=104.36.82.2:4731 , piece_id=73BDF7FRXFS5UX6R6ROROLLRZHUAYKUGNEAW3RZE3BVUZQS55DDA, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.910987600 (49h19m30.403420044s) [noise=true , address=104.36.82.6:4876 , piece_id=TSBE2UCBZ5A36IPYF3TIAMRJXVYWDWJ5KR5ZAPED737HD7VQJN3Q, seg_pos={0 0}, upload=43339]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 139.59.144.22:60622->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:27:09.911006673 (49h19m30.403439116s) [seg_pos={0 0}, upload=43339]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 31 15:27:10.020962168 (49h19m30.513394622s) [upload=43339]: Done uploading segments... Aug 31 15:27:10.020998142 (49h19m30.513430593s) [upload=43339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.375880261s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.......X.......................................] duration: 2.74337609s FAIL (2025-08-31 15:54:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.743852985s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................XXXX.......................................] duration: 3.153178357s FAIL (2025-08-31 15:53:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.15320156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XXX...XX.X...................................] duration: 3.589892948s FAIL (2025-08-31 15:53:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 31 15:30:06.929990167 (49h22m27.422422622s) [upload=43384]: Starting upload Aug 31 15:30:06.930102351 (49h22m27.422534797s) [upload=43384]: Done starting upload Aug 31 15:30:06.930192299 (49h22m27.42262473s) [upload=43384]: Uploading segments... Aug 31 15:30:06.930494009 (49h22m27.422926463s) [upload=43384]: Got next segment. Inline: false Aug 31 15:30:06.930569509 (49h22m27.423001959s) [upload=43384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:30:06.930577850 (49h22m27.423010301s) [upload=43384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:30:07.426908595 (49h22m27.91934104s) [seg_pos={0 0}, upload=43384]: Begin upload segment... Aug 31 15:30:07.426991512 (49h22m27.919423945s) [seg_pos={0 0}, upload=43384]: Done begin upload segment. Aug 31 15:30:07.427048849 (49h22m27.919481302s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427583704 (49h22m27.920016153s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.427802778 (49h22m27.920235224s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428576908 (49h22m27.921009356s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428766183 (49h22m27.921198636s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.428927432 (49h22m27.921359878s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429673596 (49h22m27.92210604s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.429742714 (49h22m27.92217516s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.430533644 (49h22m27.922966097s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.431421859 (49h22m27.923854315s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.432812916 (49h22m27.925245369s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433254450 (49h22m27.925686905s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.433649765 (49h22m27.926082215s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.434635963 (49h22m27.927068397s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.436644671 (49h22m27.929077106s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.438195088 (49h22m27.930627588s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.439352661 (49h22m27.931785092s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:07.440510736 (49h22m27.932943185s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Uploading piece... Aug 31 15:30:08.021144465 (49h22m28.513576901s) [upload=43384]: Next returned nil segment Aug 31 15:30:08.021578152 (49h22m28.514010601s) [upload=43384]: Waiting for error group managing segments... Aug 31 15:30:08.174930452 (49h22m28.667362881s) [noise=true , address=104.36.82.10:4110 , piece_id=YJSYUBB746KM5AK6EPOFJS6QDXMEKZ3YUS5IOKLTD2KNCF5SW73Q, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.494673954 (49h22m28.987106402s) [noise=true , address=104.36.82.9:4027 , piece_id=KA2IFX3BZT2EPE3AH6KPJP5TQIREDFLGU2RI6ECT6SDZDWOJQKHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.921539006 (49h22m29.413971434s) [noise=true , address=104.36.82.60:4346 , piece_id=MCVUEIHFVR5IMVMRTY4M4DVBNEVAN5GVLHZZ5JZQXRZ6JRDQDOSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:08.995148844 (49h22m29.487581287s) [noise=true , address=77.90.31.19:4873 , piece_id=FSXXBFIADN4TPW2IHHC47I2475G43BVNMLPP2RURQ53EDRLSOUHA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.096578153 (49h22m29.589010593s) [noise=true , address=79.127.205.246:4278 , piece_id=3NEFIQHJKHBZMMDSUMXFNRCGPU6WYEZRSTNOWWAWJTRT3MS7HRSA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:09.353730500 (49h22m29.846162932s) [noise=true , address=77.90.31.18:4814 , piece_id=DI2WZMGEYSOHIJROTEQUHY5Q6OLLUMEEOFKI5MHDEOQJT4UNLTYA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.507811771 (49h22m35.000244204s) [noise=true , address=77.90.31.15:4730 , piece_id=SAO6TCDOQESTKWKEBOXY76PHTLWL763SGGFDEGL4SN7CE4BRG3GQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:14.845167001 (49h22m35.337599438s) [noise=true , address=64.49.13.138:4483 , piece_id=T47ECGX6G3QYVDTLTPSYA3XURPVL6JI6TAACN3AXITEMOKUJCBWQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:15.795710445 (49h22m36.288142875s) [noise=true , address=104.36.82.14:4230 , piece_id=HRBCSR7VPCPJDWYJ2VBIMJLD7N3BA5XNFNDOCN7L6TJDUMKJK35A, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.830085953 (49h22m44.322518388s) [noise=true , address=104.36.82.2:4750 , piece_id=YEWMC3U72IECER5A4PVCYOUTD7YGY3PHSKREBHK4N6D7ACGX6TIQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:23.998192155 (49h22m44.490624599s) [noise=true , address=104.36.82.11:4147 , piece_id=BCBQHBVD74CITKMFPVUIY6HBVUBGESVLFSEJFOQQ2ZDZA6YDKONQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:25.238002546 (49h22m45.73043498s) [noise=true , address=104.36.82.4:4826 , piece_id=JUDQKL35YFIPLXRLPLX3QQJA5APYZEGNXFNDKHFB2LYVDQOOZKUA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:30:53.958598880 (49h23m14.451031321s) [noise=true , address=104.36.82.49:4004 , piece_id=PJXTJJYRZF3WZ3OJFSEU6KIL4ZMSOUOXJUUZWE42W5A3MZRCLLYQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: <nil> Aug 31 15:31:06.255978481 (49h23m26.748410935s) [noise=true , address=104.36.82.44:4903 , piece_id=HVHXHRXV4TAWYTXUZTYA6Z55DVQG4UHGI7NJO2ZJDENPPT3BIAFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256137527 (49h23m26.74856996s) [noise=true , address=104.36.82.48:4580 , piece_id=LKCD4AVKTPQK4BVZZ4AJAT5WQEQNLTXX5ULVYFJQ3ZJJS2PD7QLA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1smEnnqdNtcMugMViWEiyBVbmymV61L195zVULsp4LG42854cB, address:104.36.82.48:4580): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256217836 (49h23m26.748650265s) [noise=true , address=104.36.82.23:4459 , piece_id=F3YUWVADSCJQQXHSSM4DYLBN3YT4A7B4D25NPTLB7DX3GBGGKWAA, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256365964 (49h23m26.748798417s) [noise=true , address=104.36.82.5:4841 , piece_id=LQCF47EPR4GJVH7SP5GJE47TNKPMBMHTXVLUVAA2VSUBDCCYRHPQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: context deadline exceeded; context deadline exceeded Aug 31 15:31:06.256386380 (49h23m26.74881881s) [noise=true , address=104.36.82.48:4567 , piece_id=HSBZAMWXSJPRCKGGRO45L7I4DAICBCA6RYYNXFUNWM42DYEPAVFQ, seg_pos={0 0}, upload=43384]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:104.36.82.48:4567): protocol: rpc: dial tcp 104.36.82.48:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:31:06.256413771 (49h23m26.748846224s) [seg_pos={0 0}, upload=43384]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 31 15:31:06.504733522 (49h23m26.997165962s) [upload=43384]: Done uploading segments... Aug 31 15:31:06.504767121 (49h23m26.997199558s) [upload=43384]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.589928884s core uplink us-select-1 us1-placement-20

1
...
5185
5186
5187
5188
5189
...
8129