Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XXXX.....................................] duration: 2.442451802s FAIL (2025-08-17 04:50:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.442492865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 2.421691161s FAIL (2025-08-17 04:50:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.42173201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XXX.....................X.....................] duration: 2.387531133s FAIL (2025-08-17 04:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.387562946s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 819.473632ms FAIL (2025-08-17 04:49:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 819.505745ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration: 3.074629861s FAIL (2025-08-17 04:49:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.074656496s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 3.050162184s FAIL (2025-08-17 04:49:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.05018706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XXXX....................................] duration: 2.435456574s FAIL (2025-08-17 04:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.435494217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 2.318008173s FAIL (2025-08-17 04:48:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.318041174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XXX.....................X....................] duration: 2.704154344s FAIL (2025-08-17 04:48:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.704216115s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 3.172494345s FAIL (2025-08-17 04:48:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.172508668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 825.13013ms FAIL (2025-08-17 04:48:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 825.18167ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration: 2.849883443s FAIL (2025-08-17 04:48:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.849942645s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XXX.....................X...................] duration: 2.568704857s FAIL (2025-08-17 04:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.568736644s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 3.006049717s FAIL (2025-08-17 04:48:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.006084254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 2.284962695s FAIL (2025-08-17 04:48:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.2849956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXXX...................................] duration: 2.473672319s FAIL (2025-08-17 04:48:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.47370234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration: 2.849715957s FAIL (2025-08-17 04:48:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.849746874s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XXX.....................X..................] duration: 2.790812955s FAIL (2025-08-17 04:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.790855002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 3.582035348s FAIL (2025-08-17 04:47:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582063307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 888.519115ms FAIL (2025-08-17 04:47:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 888.553029ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 2.527118233s FAIL (2025-08-17 04:47:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.527149713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XXX.....................X.................] duration: 2.328754642s FAIL (2025-08-17 04:47:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.328789281s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 2.420363377s FAIL (2025-08-17 04:47:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.42040222s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration: 2.746765825s FAIL (2025-08-17 04:47:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.746804499s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 3.008794436s FAIL (2025-08-17 04:46:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.00883143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 919.671126ms FAIL (2025-08-17 04:46:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 919.724306ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XXX.....................X................] duration: 2.319094351s FAIL (2025-08-17 04:46:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.31918731s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 3.003026161s FAIL (2025-08-17 04:46:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.003056929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 2.623816954s FAIL (2025-08-17 04:46:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.623846202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration: 2.860997594s FAIL (2025-08-17 04:46:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.861032168s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 2.441360805s FAIL (2025-08-17 04:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.441464845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XXX.....................X...............] duration: 2.371458047s FAIL (2025-08-17 04:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.371496004s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 2.502874921s FAIL (2025-08-17 04:46:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.502930064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 826.884166ms FAIL (2025-08-17 04:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 826.923764ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 3.481529107s FAIL (2025-08-17 04:45:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481554571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 2.284724975s FAIL (2025-08-17 04:45:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.284762419s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration: 2.897084252s FAIL (2025-08-17 04:45:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.897131548s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 3.367434248s FAIL (2025-08-17 04:45:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.367460335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 2.405954907s FAIL (2025-08-17 04:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.406075431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXX.....................X..............] duration: 2.668006762s FAIL (2025-08-17 04:45:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.668116847s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 946.109289ms FAIL (2025-08-17 04:45:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 946.158417ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 2.30175812s FAIL (2025-08-17 04:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.301792033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXX.....................X.............] duration: 2.270940181s FAIL (2025-08-17 04:44:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.270970994s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration: 2.61058566s FAIL (2025-08-17 04:44:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.610623625s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 2.630272036s FAIL (2025-08-17 04:44:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.630330323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 2.866028592s FAIL (2025-08-17 04:44:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.866063118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 360.448196ms FAIL (2025-08-17 04:44:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:44:29.755715250 (203h18m51.714812507s) [upload=211625]: Starting upload Aug 17 04:44:29.755812312 (203h18m51.714909558s) [upload=211625]: Done starting upload Aug 17 04:44:29.755923188 (203h18m51.71502042s) [upload=211625]: Uploading segments... Aug 17 04:44:29.755993156 (203h18m51.715090411s) [upload=211625]: Got next segment. Inline: true Aug 17 04:44:29.755999845 (203h18m51.715097096s) [upload=211625]: Waiting for error group managing segments... Aug 17 04:44:29.756076715 (203h18m51.715173948s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:44:29.756086532 (203h18m51.715183764s) [upload=211625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:44:29.756092593 (203h18m51.715189841s) [upload=211625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:44:29.844674822 (203h18m51.803772077s) [upload=211625]: Done uploading segments... Aug 17 04:44:29.844723603 (203h18m51.803821128s) [upload=211625]: Upload finished. err: metaclient: Too Many Requests Duration:* 360.477709ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 2.299107497s FAIL (2025-08-17 04:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.299151651s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration: 2.749603644s FAIL (2025-08-17 04:44:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.749647629s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 2.599020694s FAIL (2025-08-17 04:43:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.599075065s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)